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AN  ALTERNATIVE  FRONT  END  ANALYSIS  STRATEGY  FOR  COMPLEX 
SYSTEMS 

EXECUTIVE  SUMMARY 


Research  Requirement 

This  research  is  being  conducted  in  response  to  requests  by  the  U.S.  Anny  Air  Defense 
School  Commandant,  Ft.  Sill,  Oklahoma,  and  Director  for  Manpower  and  Personnel  Integration 
(MANPRINT),  Headquarters,  Department  of  the  Army  (HQDA)  G-l.  The  purpose  of  this 
phased  research  is  to  develop  an  alternative  front  end  analysis  (FEA)  strategy  for  new  and 
evolving,  highly  automated,  complex  systems  that  rely  upon  multilevel  command  and  control 
integration.  This  research  is  in  response  to  growing  concerns  that  current  training  and  strategies 
to  identify  that  training  are  insufficient  to  address  the  growing  complexities  of  automated 
systems.  Primary  concerns  are  oriented  on  issues  related  to  supervisory  control,  problem 
solving,  and  decision-making  within  integrated  crew-based  automated  systems.  The  intent  of 
this  initial  phase  of  the  research  is  to  identify  an  alternative  FEA  process  that  will  address  these 
specific  training  requirements  and  make  training  recommendations.  Pragmatically,  this  phase 
defines  an  alternative  FEA  strategy  for  complex  systems  that  will  next  be  applied  to  Patriot  air 
battle  management  training  as  a  use  case  to  validate  and  refine  one  of  two  FEA  alternative 
strategies. 

Procedure 

Two  alternative  FEA  strategies  to  the  traditional  task-based  approach,  underlying  much 
of  current  military  training,  were  identified  by  examining  existing  literature  pertaining  to  air 
defense  training  and  perfonnance,  FEA  approaches,  and  perfonnance  and  training  issues  and 
recommendations  for  training  complex  tasks  in  individual  and  collective  settings.  Literature  was 
evaluated  for  relevance  and  applicability  to  complex  programs  of  interest.  Relevant  research 
findings  and  recommendations  were  used  to  identify  ways  the  U.S.  Army’s  current  Systems 
Approach  to  Training  (SAT)  analysis  could  be  adapted  or  replaced  to  address  specific  training 
and  performance  issues  identified  in  the  literature.  Proposed  alternative  strategies  were  reviewed 
in  terms  of  their  applicability  for  two  U.S.  Army  Air  Defense  School  programs  of  interest, 

Patriot  and  the  emerging  Anny  Integrated  Air  and  Missile  Defense  (AIAMD)  architecture. 

Thus,  our  findings  during  this  initial  phase  are  based  on  viewing  Patriot  operations  as  a  complex 
system,  research  on  multiple  FEA  approaches,  relevant  research  on  individual  and 
team/collective  training,  and  research  examining  decision-making  in  complex  situations. 

Findings 

Patriot  research  over  the  past  nine  years  has  identified  a  number  of  operational 
performance  issues  and  associated  training  concerns.  These  concerns  can  be  broadly  categorized 
into  three  areas:  (a)  operator  expertise  using  automated  systems;  (b)  skill  decay;  and  (c)  crew 
communication  and  coordination  within  the  context  of  highly  automated  systems.  We  found  that 
training  research  directly  and  indirectly  addresses  these  areas,  particularly  in  regard  to  collective 
training,  expertise,  decision-making  and  problem  solving,  and  situational  awareness  within  the 
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context  of  highly  automated  systems.  Two  alternative  FEA  approaches  were  developed  to  match 
analytic  capabilities  to  known  air  defense  training  concerns.  The  first  proposed  approach 
combines  an  event -based  analysis  with  collective  team-based  analyses.  This  approach  provides 
the  benefit  of  incorporating  collective  perfonnance  requirements  directly  within  an  operational 
mission  event  context.  The  second  approach  combines  collective  team-based  analyses  with  an 
expertise-based  analysis.  This  approach  retains  the  benefit  of  incorporating  collective 
performance  requirements  within  traditional  task-based  strategies,  as  well  as  adding  assessments 
of  individual  performance  skill  level  requirements  for  completing  complex  tasks  in  ambiguous 
situations.  Based  on  the  results  of  this  research,  an  alternative  strategy  combining  collective 
team  and  expertise-based  analyses  was  recommended  for  the  next  phase  of  this  research. 

Utilization  and  Dissemination  of  Findings 

The  results  from  this  initial  phase  were  presented  to  senior  leaders  and  staff  from  the  U.S. 
Army  Air  Defense  School  during  a  briefing  at  Ft.  Sill,  OK,  on  13  March  2014.  The 
recommended  FEA  approach  was  approved  by  the  Air  Defense  School  for  the  next  phase  of  this 
research,  i.e.  applying  the  strategy  to  current  Patriot  missile  training.  Given  the  scope  and  depth 
of  Patriot  missile  training,  this  assessment  will  be  limited  to  air  battle  management  training.  The 
results  from  this  case  use  application  will  be  presented  in  a  separate  report  following  completion 
of  the  second  phase  of  this  research. 
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An  Alternative  Front  End  Analysis  Strategy  for  Complex  Systems 

Introduction 

Historically,  front-end  analyses  (Harless,  1968)  have  been  used  to  identify  and  analyze 
performance  requirements  and  issues  at  the  beginning  of  the  instructional  design  process.  A 
front  end  analysis  (FEA)  not  only  acts  as  a  training  needs  assessment,  but  also  identifies  when 
training  is  not  required  or  the  best  option  for  addressing  a  system  requirement  (Romiszowski, 
1981;  U.S.  Department  of  the  Army,  2004).  Although  the  scope  of  an  FEA  varies  with  the 
context  in  which  it  is  applied,  organizations  have  routinely  relied  upon  systematic,  iterative 
strategies  to  define  training  needs  and  programs. 

As  a  collection  of  analyses  conducted  prior  to  implementation  of  a  project,  an  FEA 
should  define,  especially  in  military  training  environments,  the  context  of  the  training  needs  and 
answer  a  series  of  questions  that  infonn  subsequent  training  design,  development,  and  delivery. 
Figure  1  illustrates  a  generic  FEA  process  and  the  questions  typically  addressed  through  an  FEA. 
“What  needs  to  be  trained?”  is  the  central  question  of  any  training  FEA.  Performance  gaps  (e.g., 
the  difference  between  trainee  capabilities  and  mission  requirements)  provide  the  input  to  what 
needs  to  be  trained,  while  the  answers  to  that  question  lead  to  more  specific  questions  about 
when,  where,  how,  and  by  whom  training  should  be  delivered.  Regardless  of  the  types  of 
analyses  used,  or  the  specific  questions  asked,  a  consistent  feature  of  an  FEA  is  that  the  results 
are  entirely  dependent  on  what  is  included  and  the  types  of  questions  asked.  Consequently,  the 
FEA  should  be  oriented  to  address  known  training  shortcomings  regardless  of  the  source. 
Furthermore,  the  FEA  should  be  designed  to  collect  and  analyze  data  specifically  associated  with 
those  shortcomings. 


Figure  1.  FEA  core  constructs. 
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Traditionally,  instructional  developers,  especially  within  military  organizations,  in 
general,  and  the  Army,  specifically,  have  taken  a  task-  or  topic-centered  approach  to  identifying 
and  defining  the  knowledge,  skills,  and  abilities  (KSAs)  required  to  employ  developing  systems. 
The  Army’s  ADDIE  process  (Analysis,  Design,  Development,  Implementation,  and  Evaluation), 
originally  embedded  within  the  Army’s  larger  ISD  (Instructional  Systems  Design)  and  SAT 
(Systems  Approach  to  Training)  strategies,  provides  an  excellent  example  of  a  largely  task-based 
approach  to  defining  training  requirements  (see  U.S.  Department  of  the  Army,  2004;  U.S. 
Department  of  the  Army,  2013).  Key  outcomes  of  this  process  establish  criteria  for  successful 
performance  outcomes  and  set  specific  job  performance  requirements,  expectations,  and  training 
needs  and  standards.  In  many  cases,  the  overall  challenge  addressed  in  traditional  FEAs  has 
been  one  of  defining  an  organization’s  problems  or  needs  and  then  revising  or  modifying 
existing  training  requirements,  programs,  and  products  to  fit  them.  Thus,  traditional  FEAs  often 
focus  on  individual  tasks  and  roles  with  little  regard  for  critical  decision  and  coordination  points, 
interpersonal  interactive  skills  and  requirements,  or  the  full  range  of  collective,  interdependent 
activities  inherent  in  operating  complex,  networked  systems. 

This  report  summarizes  our  attempt  to  identify  or  develop  alternative  FEA  approaches 
better  framed  to  address  the  complex  challenges  and  issues  emerging  from  the  deployment  of 
highly  complex,  increasingly  automated  systems.  Our  approach  will  examine  key  components  of 
operations  within  an  exemplar  complex  system,  the  Patriot1  ADA  system,  as  well  as  research  on 
decision-making  in  uncertain  circumstances,  crew  coordination,  team  decision-making,  cognitive 
task  management,  and  the  interaction  of  individuals  within  complex  systems.  The  Patriot 
system,  as  a  use  case,  provides  an  operational  context  for  identifying  and  understanding  the 
elements  of  complex  systems  that  require  different  approaches  to  learning  and  training  than 
defined  by  traditional  FEAs.  This  research  is  being  conducted  in  response  to  requests  by  the 
U.S.  Anny  Air  Defense  School  Commandant,  Ft.  Sill,  Oklahoma,  and  the  Director  for 
Manpower  and  Personnel  Integration  (MANPRINT),  Headquarters,  Department  of  the  Anny 
(HQDA)G-l. 

Patriot  Air  and  Missile  Defense  Training 

The  Patriot  is  a  surface-to-air  missile  defense  system  that  is  routinely  deployed 
worldwide.  Patriot  has  a  long  history  of  successful  employment  by  the  U.S.  Anny.  Since  1969, 
the  system  has  undergone  several  modifications,  which  have  expanded  its  capability  and  added 
to  its  complexity.  Originally  designed  to  counter  medium  altitude  air  threats,  it  is  now  the 
Anny’s  primary  antiballistic  missile  (ABM)  system.  Patriot  is  employed  in  the  field  through  a 
battalion  echelon  organizational  structure.  The  line  battery  is  the  basic  building  block  of  a 
Patriot  unit,  and  includes  up  to  six  missile  launchers,  a  radar,  an  Engagement  Control  Station 
(ECS),  an  Antenna  Mast  Group  (AMG),  and  a  Battery  Command  Post  (BCP).  At  the  battalion 
level,  the  Information  and  Coordination  Center  (ICC)  identifies,  prioritizes,  and  coordinates 
threat  tracks  within  its  assigned  area  by  integrating  input  from  multiple  ECSs.  Together,  the 
ECS  and  ICC  are  tasked  with  monitoring  readiness,  ordering  threats,  and  giving  priorities  to 


1  The  authors  recognize  that  PATRIOT  is  an  acronym  for  “Phased  Array  Tracking  Radar  to  Intercept  on  Target”; 
however,  the  more  commonly  found  colloquial  short  reference,  “Patriot,”  will  be  used  throughout  this  report  to  refer 
to  the  PATRIOT  missile  system  and  its  components  just  as  it  appears  in  numerous  published  articles,  reports, 
directives,  and  doctrinal  manuals. 
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radar,  among  other  responsibilities.  The  ICC  is  the  link  between  the  Patriot  ECS  and  higher 
command  structures. 

Although  a  detailed  evaluation  of  current  Patriot  training  is  beyond  the  scope  of  this 
research  effort,  observations  of  current  U.S.  Anny  ADA  School  (USAADASCH)  training 
methods  suggest  current  training  methods  are  based  on  traditional  task-based  principles 
incorporated  into  the  Army's  SAT  analysis  process.  Potential  training  limitations  can  be  gleaned 
from  sources  that  attempt  to  link  training  issues  with  mission  performance  problems.  Overall, 
researchers  familiar  with  Patriot  training  and  operations  have  concluded  that  poor  operational 
performance  and  errors  may  be  due  to  disconnects  between  institutional  training,  unit  training, 
and  current  operational  responsibilities  (Hawley,  2011).  They  also  insist  that  Patriot  training  and 
assessment  should  be  more  rigorous  in  order  to  minimize  the  frequency  of  critical  decision  errors 
(Hawley,  2007;  Brooks  et  ah,  2012). 

Judgment  errors  regarding  decisions  on  the  potential  engagement  of  an  identified  threat, 
if  left  unchecked  and  unresolved,  may  ultimately  lead  to  one  of  two  final  outcomes:  (a)  allowing 
a  hostile  weapon,  mistakenly  identified  as  a  friendly  asset,  to  penetrate  the  defenses  and  damage 
or  destroy  a  protected  asset;  or  (b)  attacking  a  friendly  asset,  mistakenly  identified  as  a  hostile 
target  (i.e.,  fratricide).  Patriot  training  research  during  the  last  nine  years  has  largely  focused  on 
the  latter  of  these  outcomes  (i.e.,  fratricide  incidents)  to  detennine  the  contributing  factors  to 
these  mission  failures.  The  factors  identified  through  this  research  can  be  organized  as: 

•  lack  of  operator  expertise  using  highly  sophisticated  systems; 

•  skill  decay;  and/or 

•  poor  crew  communication  and  coordination  within  the  context  of  highly  automated 

systems. 

These  judgment  errors  may  be  minimized  with  frequent  mission  perfonnance  in  a 
wartime  environment.  In  “peacetime”  training,  however,  practice  at  this  high  level  of  tasking  is 
often  not  achieved.  As  such,  traditional  peacetime  training  models  are  being  questioned  across 
the  military  services,  with  a  growing  call  to  decrease  initial  institutional  training  time  and  move 
personnel  into  the  field  more  quickly.  This  pressure  is  often  associated  with  cost  reduction,  but 
there  is  also  a  higher  demand  for  trained  mission-ready  Soldiers  due  to  current  and  projected 
mission  requirements. 

Operator  Expertise.  Research  examining  the  link  between  mission  performance  and 
training  concluded  that  generalized  training  failed  to  develop  the  necessary  levels  of  operator 
expertise,  focusing  instead  on  individual  task  proficiencies  (Hawley,  2009;  Hawley,  Mares,  & 
Giammanco,  2006;  Hawley  &  Mares,  2006).  Expertise  can  be  described  as  consistently  superior 
performance  on  a  specified  set  of  tasks  in  a  domain  (Ericsson  &  Charness,  1994,  cited  in 
Hawley,  2009).  Concentrating  training  on  individual  task  performance,  however,  rather  than 
whole-job  proficiency,  does  not  necessarily  result  in  expertise  (Schneider,  1985,  cited  in  Hawley, 
2009).  The  use  of  rote  practice  and  routine  drills  of  system  processes,  such  as  typical  in  Patriot 
training,  reinforces  automation  bias,  which  is  the  bias  towards  accepting  outcomes  from 
automation  (Hawley,  2007).  Automation  bias  can  result  in  failures  of  monitoring  (vigilance 
problems)  and  decision  biases,  whereby  the  operator  automatically  defers  to  the  machine  output. 
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Over-reliance  on  the  machine  may  be  directly  attributed  to  a  lack  of  expertise  in  understanding 
how  the  system  actually  operates  and  when  circumstances  deviate  from  standard  procedures  and 
training.  The  disconnect  between  task-based  training  and  expert  perfonnance  can  be  traced  back 
to  how  training  requirements  are  detennined,  as  well  as  how  training  outcomes  are  measured. 

Related  to  automation  bias  and  individual  task  proficiency  training,  Hawley  (2007) 
concluded  that  operators  lack  an  adequate  comprehension  of  the  tactical  situation.  He  attributed 
this  to  Patriot  system  operators  being  trained  to  successfully  repeat  and  complete  routine  drills 
rather  than  actively  construct  a  mental  model  of  the  tactical  situation  and  its  relevant 
components.  The  failure  to  frame  an  accurate  mental  model  and  understanding  of  the 
operational  and  tactical  environment  constrains  operator  responses  in  novel  situations  and 
scenarios.  If  whole-job  proficiency  is  an  aspect  of  expertise,  then  training  to  a  more  advanced 
understanding  of  the  tactical  situation  and  effective  adaptation  to  change  needs  to  build  on  a 
comprehension  of  system  operation  and  individual  task  proficiencies.  Hawley  and  Mares  (2007) 
and  Brooks  et  al.  (2012)  concluded  that  adaptive  decision-making  is  not  enabled  among  Patriot 
operators  (nor  trainees),  and  that  training  should  focus  less  on  routine  drills  and  more  on  the 
application  of  varied  scenario-based  simulations  to  encourage  decision-making  in  complex 
situations. 

Skill  decay,  Hawley  and  Mares  (2006)  assert  that  high  levels  of  technological 
automation  will  contribute  to  skill  decay  (i.e.,  forgetting)  in  Soldiers,  as  well  as  other 
performance  issues,  as  they  increase  their  reliance  on  automated  processes  and  decrease  their 
reliance  on  their  own  knowledge  and  expertise.  Citing  Klein  (2003),  Hawley  and  Mares  caution 
that  increasing  automation  can  impair  individual  skills  in  three  ways  beyond  basic  skill  decay: 

(a)  disabling  the  expertise  of  skilled  operators;  (b)  slowing  the  rate  of  learning  enough  to  block 
the  development  of  systemtic  expertise;  and  (c)  reinforcing  dysfunctional  skills  that  interfere 
with  subsequent  expertise  attainment. 

According  to  Brooks  et  al.  (2012),  the  Institute  of  Defense  Analyses  (IDA)  concurred 
with  Hawley  and  Mares’  conclusions,  predicting  that  “based  on  the  task  description  for  the 
original  Patriot  air  battle  engagement,  the  perfonnance  of  operators  will  diminish  quickly  after  a 
period  of  nonuse  as  predicted  by  the  Hagman-Rose  [see  Hagman  and  Rose,  1983]  skill  retention 
model”  (Brooks  et  al.,  2012,  p.  1 1).  Left  unchecked,  these  effects  will  eventually  result  in  a 
small  percentage  of  operators  sufficiently  skilled  to  perfonn  the  task  following  a  retention 
interval  or  a  period  of  nonuse.  Brooks  and  colleagues  further  point  out  that  successfully  retained 
skills  may  be  limited  and  may  not  improve  with  practice  unless  the  training  adds  novelty  and 
challenge  to  the  scenarios.  Skill  decay  and  the  over-reliance  on  automation  features  will 
negatively  affect  mission  performance  and  training  progress  unless  such  effects  are  addressed 
directly  through  training  and  training  designs. 

System  technology.  The  appropriate  level  of  automation  and  human  interaction  within 
technologically  advanced  operational  systems  is  a  common  topic  in  the  field  of  human  factors, 
and  automation  in  Patriot  is  no  exception  (Endsley,  1997a;  Hawley,  2011).  Fratricide  incidents 
during  Operation  Iraqi  Freedom  pushed  this  issue  to  the  forefront  of  Patriot  design  and  training. 
The  Defense  Science  Board  (DSB)  created  a  task  force  to  examine  the  lessons  from  the 
Operation  Iraqi  Freedom  Patriot  systems’  perfonnance  and  detennine  how  future  development  of 
the  Patriot  or  follow-on  systems  could  avoid  future  incidents  (Brooks  et  al.,  2012).  The  DSB 
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summary  report  declared  that  the  operating  protocol  was  largely  automatic,  and  operators  were 
trained  to  trust  the  information  generated  by  the  system’s  software.  The  solution  proposed  to 
address  deficiencies  noted  in  the  fratricide  incidents  was  “more  operator  involvement  and  control 
in  the  functioning  of  the  Patriot  battery,  which  will  necessitate  changes  in  software,  displays,  and 
training”  (Defense  Science  Board,  2005,  p.  2,  as  cited  in  Brooks  et  al.,  2012,  p.  17).  This 
recommendation  focused  on  Patriot  system  designs  rather  than  significantly  altering  Patriot 
training.  The  technology,  however,  received  similar  examination  in  tenns  of  how  to  train  to  the 
current  automation-human  balance. 

Concurrent  to  the  DSB  inquiry,  Hawley  and  associates  examined  the  impact  of 
automation  on  air  defense  command  and  control  operators  and  the  consequences  of  changing 
their  roles  from  traditional  operators  to  supervisors  of  automated  processes  (Hawley,  Mares,  & 
Giammanco,  2005).  As  explained  by  Hew  and  colleagues,  “supervisory  control  is  where  a 
machine  closes  a  control  loop,  and  a  supervisor  intermittently  programs  the  machine”  (Hew, 
Lewis,  Radunz,  &  Rendell,  2010,  p.  1).  At  issue  is  how  two  decades  of  technical  enhancements 
through  system  evolutions  have  changed  operator  roles  from  being  traditional  duties  to 
supervisors  of  automated  processes  (Hawley,  Mares,  &  Giammanco,  2005).  Training,  however, 
did  not  keep  up.  “Operators  lacked  the  rigorous  training  to  deal  with  this  different  role  —  a  role 
as  supervisor  of  automated  functions  and  services  subordinate  to  the  operator”  (Brooks  et  al., 
2012,  p.  5).  Brooks  and  his  associates  concluded  that  training  shortcomings  are  the  result  of  a 
lack  of  proper  job  task  analysis  (focusing  on  system  operation  tasks  rather  than  supervisory 
controls  tasks),  the  lack  of  training  requirements  reanalysis  during  or  after  significant  upgrades, 
and  a  reliance  on  training  devices  that  focused  on  narrow  training  scenarios  or  rote  training 
methods.  These  issues  are  a  direct  consequence  of  the  continual  evolution  and  technological 
automation  advances  of  the  Patriot  system.  Unless  addressed  by  new  training  designs,  these 
issues  are  likely  to  be  compounded  with  the  air  defense  community’s  pending  transfonnation  to 
the  Army  Integrated  Air  and  Missile  Defense  (AIAMD)  architecture. 

In  moving  forward  with  the  AIAMD  architecture,  the  Anny  is  developing  and  testing 
new  air  and  missile  defense  (AMD)  systems  integrating  air  defense  artillery  (ADA)  sensors  and 
shooters  under  a  common,  networked  battle  command  architecture.  A  critical  component  of  this 
architecture  is  the  AIAMD  Battle  Command  System  (IBCS)2.  The  IBCS  comprises  hardware 
and  software  pennitting  AIAMD  enabled  sensors  and  weapons  platforms  to  be  engaged  remotely 
via  an  integrated  fire  control  network  (IFCN)  operated  from  IBCS  Engagement  Operations 
Centers.  Each  sensor  and  weapons  platform  will  have  a  plug  and  fight  interface  module 
providing  distributed  battle  management  functionality  enabling  network-centric  operations 
through  the  IBCS.  Although  originally  designed  to  integrate  a  wide  range  of  established  and 
evolving  systems,  recent  decisions  regarding  developing  and  fielding  a  number  of  these  systems 
have  focused  resources  on  the  integration  of  the  Patriot  missile  system  into  the  AIAMD  and  its 
supporting  IBCS.  Thus,  this  architecture  employs  an  overarching  system-of-systems  capability 
with  participating  AMD  components  functioning  interdependently  to  provide  a  level  of 
operational  capability  not  achievable  by  the  individual  systems  acting  alone.  Many  leaders  and 


2  Although  “IBCS”  technically  refers  to  the  Army’s  Integrated  Air  and  Missile  Defense  (AIAMD)  Battle  Command 
System,  it  most  often  is  referred  to  by  its  shorter  acronym,  Integrated  Battle  Command  System  (IBCS),  to  limit 
confusion  when  discussed  in  tandem  with  the  overarching  system  architecture,  AIAMD.  The  shorter  referent  will  be 
used  throughout  the  remainder  of  this  report  for  the  same  reasons. 
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training  experts  are  concerned  that  current  training  practices  and  standards  will  not  meet  the 
challenge  of  developing  expertise  in  these  types  of  complex,  technology-intensive  systems  and 
architectures  (see  Hawley,  2011). 

While  examining  the  Patriot  missile  system  can  enhance  our  practical  understanding  of 
the  types  of  tasks  and  decision-making  required  for  complex  systems,  the  soon  to  be  deployed 
AIAMD  architecture  adds  a  greater  level  of  complexity  to  these  requirements.  The  AIAMD 
architecture  will  integrate  various  AMD  sensors  and  weapons  into  a  single,  integrated  fire 
control  and  battle  command  network.  This  integration  within  a  system-of-systems  design  will 
provide  a  level  of  operational  capability  not  readily  achievable  by  these  individual  systems 
operating  independently  within  a  theater  of  operations.  It  is  anticipated  that  the  integration  of 
these  systems  will  significantly  increase  the  challenges  Soldiers  face  in  monitoring  and 
defending  an  increasingly  dynamic  and  expanded  battlespace,  particularly  in  the  areas  of  sensor 
operations  and  communication,  command,  and  control  functions.  An  additional  challenge  will 
be  a  lengthy  transition  period,  during  which  Soldiers  will  be  required  to  manage  and  work  in  a 
mixed  environment  of  transitioning  IBCS  enabled  and  legacy  systems. 

Based  on  many  of  the  issues  highlighted  by  Hawley  and  his  colleagues  (see  Hawley, 
2011;  Hawley  &  Mares,  2006;  Hawley,  Mares,  &  Giammanco,  2006),  the  impact  of  combining 
multiple  air  defense  operations,  sensors,  and  weapon  systems  in  an  IAMD  environment  could 
potentially  overwhelm  decision  makers  at  each  level  of  the  engagement  process.  Not  only  will  it 
be  critical  to  ensure  the  software  and  systems  perfonn  as  designed,  but  careful  consideration 
must  also  be  paid  to  the  projected  workload  demands  and  training  required  in  both  institutional 
and  unit  training  environments  to  prepare  Soldiers,  and  their  instructors,  to  effectively  carry  out 
their  responsibilities  in  such  a  complex,  overlapping  battlespace.  Since  traditional  FEA 
strategies  have  largely  been  task  focused  processes  designed  for  fairly  well-defined  or  bounded 
operational  environments,  they  seem  increasingly  ill-suited  for  progressively  more  complex, 
multisystem  environments  that  significantly  alter  the  roles  of  human  decision  makers  and 
operators  within  highly  automated  command  and  control  environments. 

Method 

As  described  earlier,  the  overall  goal  of  this  research  is  to  develop  new  FEA  strategies  for 
new/evolving,  highly  automated,  complex  systems  relying  on  multilevel  command  and  control 
integration.  In  order  to  establish  an  operational  context  for  this  multiphase  research  effort,  it  will 
focus  on  current  Patriot  system  training  and  its  evolving  transition  to  the  AIAMD  architecture. 
This  initial  phase  of  the  research  will  identify,  propose,  and  justify  alternative  FEA  strategies  for 
complex,  highly  automated  systems,  in  general,  and  the  Patriot  missile  system,  specifically. 
Subsequent  phases  of  this  research  will  apply  the  strategy  recommended  at  the  end  of  this  report 
to  identify  potential  gaps  in  current  Patriot  training  and  certification  requirements  and,  later,  to 
Patriot’s  transition  into  the  AIAMD  architecture. 

In  order  to  reach  the  goals  of  this  research,  the  research  team  reviewed  published 
literature  on  FEA  strategies,  processes,  and  applications.  The  team  also  conducted  an  extensive 
review  of  research  on  the  challenges  and  issues  associated  with  complex,  highly  automated 
systems,  with  a  specific  focus  on  issues  impacting  their  design  and  application  to  various  training 
courses  or  programs  of  learning.  Finally,  detailed  infonnation  was  collected  and  examined  from 
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training  units  and  weapon  system  subject  matter  experts  (SMEs)  regarding  current  Patriot  system 
components  and  processes,  training  systems,  and  perfonnance  requirements. 

Literature  Findings 

Building  upon  initial  literature  sources  and  published  reports  provided  by  ARI 
researchers,  the  team  identified  additional  relevant  literature  through  an  examination  of 
references  cited  in  these  initial  sources  and  through  online  searches  of  library  research 
catalogues.  This  resulted  in  a  compilation  of  peer-reviewed  articles,  industry  association 
publications,  government  documents,  books,  and  conference  proceedings  focusing  on  FEA, 
training  design  and  development,  as  well  as  identified  Patriot  training  issues  and  challenges. 

This  literature  was  examined  to  detennine  the  applicability  and  appropriateness  of  the 
reported  FEA  research  to  this  specific  research.  The  criteria  used  by  the  team  to  evaluate  the 
merits  of  these  works  included  the  publication’s  discussion  of  complex,  integrated  command  and 
control  systems;  balance  of  human  operator  and  command  automation  roles  and  requirements 
within  complex,  highly  automated  systems;  and  ADA  training  requirements,  issues,  and 
strategies.  The  extent  to  which  the  infonnation  could  be  generalized  to  the  ADA  community  was 
also  considered.  Specifically,  the  team  determined  if  and  how  the  literature  addressed  the 
following  domain  areas: 

•  existing  and  emerging  FEA  models  and  processes; 

•  human-machine  interactions; 

•  situational  awareness; 

•  individual  and  collective  tasks; 

•  multiple  levels  of  performance; 

•  decision  points; 

•  routine  and  adaptive  decision-making; 

•  cognitive  tasks; 

•  the  operating  environment;  and 

•  other  issues  identified  by  publications  that  emerged  as  recurring  themes. 

The  team  initially  intended  to  analyze  existing  FEA  models  and  potential  new  models 
that  could  address  the  complexities  of  the  identified  training  systems  and  issues  identified  in  the 
literature.  During  the  course  of  the  literature  review,  however,  it  became  evident  that  this  body 
of  literature  did  not  include  specific,  well-defined  FEA  models  that  could  be  considered  as 
alternatives  to  the  established,  traditionally  task-based  approaches  at  the  core  of  military  training 
designs.  Consequently,  the  team  focused  on  analytic  models  and  alternatives  that  addressed,  or 
potentially  could  address,  the  entire  instructional  design  process  from  beginning  to  end.  The 
goal  of  this  review  was  to  identify  components  that  could  be  adapted  in  whole  or  in  part  to  meet 
our  project  goals. 

The  following  discussion  summarizes  the  team’s  evaluation  of  a  number  of  model 
components  and  characteristics  based  on  multiple  sets  of  criteria,  including  the  extent  to  which 
they  addressed  issues  of  interest  to  the  ADA  community,  e.g.,  human-machine  interactions, 
situational  awareness,  adaptive  decision-making,  collective  tasks,  critical  decision  points,  and  a 
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military  operating  environment.  FEA  process  issues,  such  as  data  requirements,  time,  and 
expertise  required  for  applying  the  strategy,  are  also  considered  in  this  discussion. 

FEA  Approaches  and  Design 

U.S.  Army  System  Approach  to  Training  Analysis.  The  Department  of  Defense 
(DOD)  provides  guidance  for  the  development  of  instructional  systems  through  the  DOD 
military  handbook  five-part  series,  MIL-HDBK  29612-1A,  -2A,  -3A,  -4A,  and  -5,  published  31 
Aug  0 1 ,  where  each  volume  focuses  on  a  unique  aspect  of  training  acquisition  and  Instructional 
Systems  Design  (ISD).  MIL-HDBK-29612-2A,  Instructional  Systems  Development/Systems 
Approach  to  Training  and  Education  (Part  2  of  5  Parts),  addresses  DOD’s  system  approach  to 
training  and  education  (i.e.  SAT)  (U.S.  Department  of  Defense,  2001).  The  DOD  ISD  process 
follows  the  cyclical  “PADDIE”  model.  “PADDIE”  is  an  acronym  for  planning,  analysis,  design, 
development,  implementation,  evaluation.  The  complete  DOD  ISD/SAT  model  is  shown  in 
Figure  2,  as  adapted  from  MIL-HDBK  29612-2A  guidance  (U.S.  Department  of  Defense,  2001, 

p.  10). 


QUALITY  MANAGEMENT  IMPROVEMENT 


ADMINISTRATION 


Figure  2.  DOD  ISD/SAT  model. 

Each  branch  of  the  U.S.  military  adapts  DOD  guidance  to  some  degree  in  order  to  match 
their  specific  requirements,  doctrine,  and  language.  The  Army’s  approach  to  SAT  is  documented 
in  a  series  of  Training  and  Doctrine  Command  (TRADOC)  pamphlets;  TRADOC  Pamphlet  350- 
70-6  (U.S.  Department  of  the  Army,  2004)  presents  the  analysis  phase  of  SAT. 

The  FEA  portion  of  the  Army’s  SAT  includes  five  types  of  training  analyses:  needs, 
mission,  collective  task,  job,  and  individual  task  analyses.  The  analysis  process  flow  and 
relationships  are  shown  in  Figure  3.  Like  other  DOD  Services,  the  Army’s  SAT  process  utilizes 
a  variety  of  analyses,  which  may  or  not  be  directly  applicable  to  any  specific  project.  For 
example,  a  training  system  evaluation  may  trigger  the  need  for  a  requirements  analysis  beginning 


with  a  mission  or  job  analysis,  without  being  part  of  or  included  in  an  actual  FEA.  Similarly,  the 
project’s  scope  can  determine  whether  collective  or  just  individual  task  analyses  will  be 
conducted. 


Conduct  Needs 


Training 


Figure  3.  U.S.  Army  SAT  analysis  process  (U.S.  Army,  2004,  p.  10). 

As  briefly  described  earlier  in  this  report,  the  Army’s  SAT  analysis  is  primarily  a  task- 
based  analysis  process.  In  other  words,  tasks  are  the  primary  unit  of  analysis,  regardless  if  those 
tasks  are  cognitive  or  psychomotor  in  nature.  As  shown  in  Figure  4,  the  mission  is  deconstructed 
into  jobs,  which,  in  turn,  are  deconstructed  into  tasks.  Tasks,  in  turn,  are  deconstructed  into  sub¬ 
tasks  and  steps.  Tasks,  however,  are  the  basis  for  determining  learning  objectives  and  training 
strategies  in  subsequent  analyses.  Task  attributes,  such  as  performance  difficulty  and  required 
knowledge  or  skill  learning  levels,  infonn  decisions  about  training  methods  and  delivery  options. 
Tasks  in  a  task-based  analysis  represent  the  intersection  between  what  to  train  (deconstructed 
from  mission  performance)  and  how  to  train  (based  on  specific  task  attributes). 


Intersection  between  Mission  parameters/ 
requirements  and  how  it  should  be  trained 


Figure  4.  Mission-task  deconstruction. 


This  process  provides  for  a  step-wise  standardized  procedure  that  may  be  applied 
regardless  of  specific  missions  or  jobs.  While  the  task-based  process  provides  a  logical  linkage 
between  mission  requirements  and  training  tasks,  the  deconstructive  nature  of  the  analysis  also 
results  in  some  weaknesses.  Task  deconstruction  usually  results  in  the  tasks  being  analyzed  out 
of  context.  One  consequence  is  that  crew  and/or  system  coordination  is  insufficiently  analyzed. 
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The  standard  process  identifies  collective  tasks,  but  orients  analysis  on  individual  perfonnance 
rather  than  a  systems  understanding.  This  yields  an  identification  of  tasks  that  must  be 
coordinated  but  fails  to  identify  or  analyze  the  coordination  processes  necessary  for  successful 
mission  performance.  A  second  consequence  is  that  higher  level  cognitive  requirements  that 
reflect  task  context,  such  as  situational  awareness  or  crew  resource  management  (CRM),  are 
often  ill-defined,  which  results  in  ill-defined  training  requirements  and  training 
recommendations. 

A  review  of  Patriot  training  documentation,  research,  and  training  observations  supported 
our  earlier  assertion  that  current  Patriot  training  was  based  on  the  Army’s  SAT  task-based 
analysis  to  determine  training  requirements.  While  this  approach  provided  a  comprehensive 
framework  for  detennining  and  structuring  Patriot  training  tasks,  it  also  contains  a  focus  on 
individual  task  perfonnance  with  a  minimal  understanding  of  how  the  automated  system  may 
affect  Soldiers’  cognitive  interpretation  of  mission  events  or  communication  and  coordination 
among  crew  members. 

Given  the  limitations  of  the  current  SAT  approach,  it  was  detennined  that  alternative 
analytic  approaches  should  be  considered.  The  current  approach  does  not  seem  to  adequately 
address  interactions  within  complex  systems.  Secondly,  the  current  approach,  while  task-based, 
focused  more  on  the  physical  tasks  required  to  operate  a  system  rather  than  the  complex 
cognitive  tasks  required  to  successfully  complete  a  mission.  Last,  the  current  approach  focuses 
on  individual  performance,  while  the  perfonnance  of  more  complex  missions  such  as  required 
for  Patriot  operations  require  integrated  performance  as  a  member  of  a  crew/team. 

Alternative  FEA  Considerations.  The  DOD  ISD  process,  in  general,  and  the  U.S. 
Army’s  SAT  analysis  process,  specificially,  focus  on  tasks  required  for  successful  operation  of 
systems  and  missions.  Army  organizations  conduct  FEAs  to  support  course  design, 
development,  implementation,  and  evaluation.  While  specific  analytic  models  and  processes 
may  vary  across  projects  and  organizations,  the  ISD  guidelines  established  by  the  DOD  and  U.S. 
Army  set  standards  that  must  be  adhered  to  when  designing  and  structuring  training  programs. 
This  research  provides  an  opportunity  to  identify,  explore,  and  test  alternate  methods  for 
conducting  FEAs.  It  is  our  opinion  that  a  complementary  FEA  approach  (i.e.,  one  that  leverages 
existing  Army  task-based  processes  and  terminology)  is  most  desirable  for  framing  FEA 
applications  to  future  training  situations. 

Variation  in  how  or  what  analyses  are  conducted,  depending  on  project  characteristics, 
needs,  or  goals,  is  routine  and  within  the  bounds  of  established  ISD  guidelines  and  standards. 

For  example,  in  the  conduct  of  a  large-scale  complex  U.S.  Navy  aviation  FEA,  Weeks,  LeVita, 
and  Hadley  (2008)  incorportated  a  bottom-up  analysis  (i.e.,  categorizing  tasks,  duties,  and  tasks 
based  on  task  perfonnace)  to  supplement  the  usual  top-down  analysis  (i.e.,  mission-duty-tasks) 
employed  by  the  Navy.  Likewise,  Buehner,  Drzymala,  and  Brent  (2010)  incorporated  a 
multilevel  analysis  to  support  the  identification  and  delineation  of  complex  tasks  associated  with 
large  crew  mission  perfonnance. 

The  literature  on  alternative  FEAs  is  limited.  There  appears  to  be  little  impetus  and 
opportunity  for  practioners  (Government  or  otherwise)  to  aggressively  seek  or  develop  new 
approaches  due  to  the  requirements  of  established  DOD  standards  and  processes.  Despite  the 
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historical  success  of  the  current  DOD  FEA  processes,  there  is  growing  concern  that  these 
processes  are  insufficient  to  address  emerging  analytical  needs  for  complex  tasks  and  the  way 
modem  warfare  is  conducted  (see  Hawley,  2011;  Hawley  &  Mares,  2006;  Hawley,  Mares,  & 
Giammanco,  2006).  Task  complexity  is  increasing  as  the  technology  of  tools  used  by  Soldiers 
advances.  Individual  Soldiers  can  accomplish  more  with  advanced  tools;  but  doing  more 
translates  into  increased  information  processing  demands,  situational  awareness  requirements, 
decision-making,  communication,  and  coordination.  Complex  tasks  that  are  difficult  to 
deconstruct  and  analyze  using  standard  FEA  processes  can  be  roughly  categorized  as  individual 
information-processing,  situational  awareness,  and  adaptive  decision-making;  and  collective 
(team)  communication  and  crew  management. 

In  a  study  of  best  training  design  practices,  Visscher-Voerman  and  Gustafson  (2004) 

3 

examined  how  expert  designers,  as  identified  by  their  peers,  implemented  the  ADDIE  model. 

The  strategies  of  24  expert  professional  designers  were  examined,  including  the  types  of  analytic 
models  incorporated  into  their  front  end  analyses.  While  this  study  did  not  focus  on  DOD 
practices,  the  most  common  type  of  analysis  utilized  by  these  experts  was  task  analysis  (see 
Table  1).  This  study  illustrated  that  FEAs  are  not  fixed  processes,  but  employ  a  variety  of 
analyses  dependent,  presumably,  on  project  needs. 


Table  1. 

Types  of  “ADDIE”  Analyses 


Analysis  Type 

Number  of  Designers 
Using  (AM 24) 

Problem  analysis 

3 

User  needs  assessment 

6 

Reuse  or  repurpose  analysis 

5 

Prototype  design  and  analysis 

4 

Solution  analysis 

6 

Content  analysis 

6 

Task  analysis 

8 

Feasibility  study 

5 

Cognitive  task  analysis  (CTA)  is  a  specialized  approach  that  attempts  to  capture  and 
understand  cognitive  processes  that  underlie  behavior  that  may  be  used  to  supplement  or  replace 
“normal”  task  analysis  (see  Chipman,  Schraagen,  &  Shalin,  2000;  Clark  &  Estes,  1996).  Like 
other  types  of  analyses,  CTA  can  be  integrated  within  a  FEA  approach.  CTA  is  not  a  single 
approach,  but  represents  a  collection  of  more  than  100  methods  and  processes  (Yates,  2007). 
Although  specific  processes  vary,  all  assume  that  understanding  comes  from  information 
provided  by  experienced,  proficient  experts  in  the  domain  under  study  (Munro  &  Clark,  2013). 
While  CTA  exists  as  a  separate  field  of  study,  its  principles  and  practices  are  often  incorporated 
into  task  analysis  procedures  in  common  practice.  For  example,  the  term  “behavioral”  task  has 
been  generally  replaced  with  “psychomotor”  task,  an  acknowledgement  that  most  tasks  are  a 
combination  of  cognitive  process  and  behavioral  actions.  However,  CTA  offers  a  more 
cognitively  focused  data  framework  that  can  be  incorporated  into  an  alternative  FEA  process. 


3  This  model  assumes  that  planning  is  an  interative  and  continuous  component  of  the  core  ADDIE,  thus  planning  is 
not  identified  as  a  separate  process. 
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An  alternative  FEA  model  presented  by  Gordon  (1991)  provides  a  framework  loosely 
based  on  three  levels  of  human  cognition:  implicit  processing,  rule-based,  and  analytic. 

Gordon's  proposed  approach  is  oriented  to  the  development  of  artificial  intelligence  systems  and 
is  subsequently  strictly  cognitive  in  nature.  The  lesson  of  Gordon's  research  is  that  an  FEA  must 
gather  infonnation  relevent  to  certain  factors,  such  as  cognitive  characteristics,  and  use  that 
infonnation  to  guide  subsequent  development.  This  approach  is,  of  course,  common  to  most 
FEA  strategies  and  offers  little  guidance  for  military  training  system  designs. 

Van  Merrienboer  and  collegues  (2003)  detailed  an  instructional  design  approach  oriented 
toward  complex  learning  according  to  Cognitive  Load  Theory.  This  Four-Component 
Instructional  Design  model  (4C/ID)  (see  also  van  Merrienboer,  Clark,  &  de  Croock,  2002) 
couples  complex  problem  solving  with  completing  procedures  matching  an  authentic  set  of  tasks. 
It  incorporates  common  training  principles  such  as  scaffolding,  whole  task  practice,  and  just-in- 
time  learning  to  allow  learners  to  navigate  an  increasingly  complex  learning  environment 
through  their  own  increasing  skill-sets  and  just-in-time  external  support  that  provides 
infonnation  on  performance  and  knowledge  requirements.  One  of  the  key  discriminating 
elements  of  van  Merrienboer  et  al.’s  model  was  the  identification  of  appropriate  part-tasks  that 
supports  very  complex,  whole-task  learning.  This  approach  required  that  sufficient  information 
be  gathered  to  inform  the  four  principle  components  of  the  model: 

1.  learning  tasks  (i.e.,  the  learning  tasks  need  to  be  identified); 

2.  supportive  information  (i.e.,  the  knowledge  or  information  needed  to  perform  each  task); 

3.  procedural  information  (i.e.,  the  steps  and  processes  necessary  to  performance  each  task); 

and 

4.  part-task  practice  (i.e.  the  constituent  elements  of  each  task). 

The  authors  also  differentiate  between  variable  (e.g.,  reasoning  and  decision-making)  and 
recurrent  (e.g.,  routine)  tasks,  which  each  suggest  different  types  of  training  approaches. 

In  a  recent  review  of  training  efficacy,  Vogel-Walcutt  and  colleagues  (2013)  concluded 
that  technology-based  experiential  learning  approaches  are  the  best  ways,  to  date,  to  train 
military  adaptation,  decision-making,  and  problem  solving.  The  authors  argued,  however,  that 
most  training  systems  are  best  described  as  “practice  platforms”  rather  than  training  devices,  due 
to  insufficient  use  of  instructional  design  principles  in  the  original  design  of  the  systems.  The 
appropriate  use  of  instructional  design  principles  that  match  design  considerations  to  appropriate 
instructional  methods  and  media  must  be  supported  by  sufficient  and  appropriate  data  collected 
during  an  FEA.  Platforms  intended  to  train  adaption,  decision-making,  and  problem  solving 
must  provide  trainees  with  opportunities  to  try,  fail,  and  learn,  rather  than  simply  practicing 
routine  or  predictable  tasks.  The  FEA,  then,  must  identify  and  analyze  real  world  situations  and 
environments  that  require  more  complex  behaviors  and  skills. 

Despite  interest  in  altering  FEA  models  to  better  address  more  cognitive,  advanced,  and 
crew-based  training  tasks,  alternative  FEA  models  and  processes  and  processes  have  not  gained  a 
significant  foothold  in  the  training  community.  Contrary  to  the  well  defined  precedence  (and 
requirements)  of  the  standardized  DOD  model,  alternative  approaches  are  typically  steeped  in 


12 


theory  and  lack  pragmatic  instruction  for  non-expert  practitioners.  A  challenge,  therefore,  is  to 
identify  alternative  approaches  that  can  be  framed  for  practical  application. 

Collective  Training 

One  basic  FEA  question  is  “Who  needs  to  be  trained?”  This  question  is  often  answered 
before  the  analysis  is  begun.  The  scope  of  an  FEA  is  typically  defined  by  a  personnel 
designation,  which  is  then  used  to  further  define  associated  mission,  jobs,  and  duties.  A  potential 
shortcoming  of  this  standard  approach  is  that  it  frequently  removes  the  individual  Soldier  from 
the  context  of  what  needs  to  be  trained.  In  other  words,  a  Soldier's  individual  mission,  jobs, 
duties,  and  tasks  are  prioritized  above  the  unit's  mission,  and  collective  tasks  are  conceptualized 
as  an  off-shoot  of  an  individual's  task  performance.  The  mission,  not  specific  personnel  or  task 
defined  positions,  should  be  the  detennining  factor  in  deciding  “who”  should  be  included  in  the 
FEA.  In  the  case  of  Patriot,  it  is  quite  clear  that  demands  placed  upon  the  entire  Patriot  crew 
should  be  carefully  considered. 

Teamwork  and  teamwork  skills  are  closely  related  to  expertise.  Kozlowski  (1998)  found 
that  expertise  is  best  developed  within  intact  teams  operating  in  realistic  performance  settings. 
Intact  teams  are  those  teams  that  remain  together  to  work  as  a  crew,  rather  than  the  team 
membership  changing  for  each  mission.  Following  this  teamwork  orientation,  Hawley  and 
Mares  (2007)  suggest  borrowing  from  crew  resource  management  (CRM)  research  and 
strategies.  As  described  by  Helmreich  and  Foushee  (1993),  CRM  focuses  on  crew-level,  rather 
than  individual  level  training  and  performance  processes  and  outcomes.  CRM  seeks  to  optimize 
“. .  .not  only  the  person-machine  interface  and  the  acquisition  of  timely,  appropriate  information, 
but  also  interpersonal  activities  including  leadership,  effective  team  formation  and  maintenance, 
problem-solving,  decision-making,  and  maintaining  situation  awareness”  (Helmreich  and 
Foushee,  1993,  p.  4).  While  the  term  has  been  most  commonly  applied  to  aviation  disciplines, 
CRM  provides  a  proven  framework  for  identifying,  managing,  and  developing  crew-based 
processes  and  proficiencies.  Hawley  and  Mares  (2007)  identified  a  number  of  common  CRM 
training  elements  that  aligned  to  behavior  categories  directly  relevant  to  Patriot  crew 
performance: 

•  mission  planning  and  evaluation; 

•  task  management; 

•  enhancing  situational  awareness; 

•  crew  coordination; 

•  communications; 

•  risk  management;  and 

•  tactics  employment. 

What  is  important  to  consider  is  that  each  of  these  CRM  proficiencies  reflects  well- 
defined  collective  or  team  behaviors  and  outcomes,  and  related  perfonnance  areas  for  individuals 
operating  complex  systems.  Although  each  member  of  the  crew  has  unique  individual 
responsibilities  that  contribute  to  the  whole,  the  crew’s  performance  is  a  collective  outcome  that 
requires  more  than  the  simple  aggregation  of  isolated  individual  outcomes  within  a  standardized 
perfonnance  sequence.  For  example,  if  communication  and  coordination  between  crew 
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members  breaks  down  and  hinders  the  transfer  of  critical  information  within  the  crew,  it  is 
difficult  to  see  how  the  crew,  or  the  appropriate  command  positions,  can  arrive  at  a  correct 
decision  or  expediently  resolve  an  issue.  This  communication  transfer,  often  critical  in  aviation 
circumstances,  is  also  relevant  to  situations  such  as  the  Patriot.  Communication  among  the  crew 
and  to  higher  authorities  is  critical  to  successful  mission  perfonnance.  Through  observation,  this 
communication  appears  most  successful  when  common  team  tenninology  is  established  within 
the  intact  team. 

Kozlowski  and  Klein  (2000)  provide  a  useful  framework  for  differentiating  between 
individual,  collective,  and  other  team  based  requirements.  Team  members  engage  in  task  work 
and  teamwork  processes.  Individual  task  work  elements  are  the  performance  components  that  do 
not  require  interaction  with  other  team  members.  Teamwork,  on  the  other  hand,  encompasses 
both  the  interdependent  components  that  must  be  perfonned  by  multiple  individuals  (i.e., 
collective  tasks),  as  well  as  the  behaviors  within  the  team  that  promote  high  performance  (e.g., 
good  communication  and  coordination).  Team  perfonnance  is  thus  the  result  of  individual  task 
work,  collective  tasks,  and  the  interaction  of  the  team  itself  that  must  be  managed  and 
coordinated  to  achieve  successful  outcomes.  Thus,  teamwork  can  be  defined  as  “the 
interdependent  components  of  performance  required  to  effectively  coordinate  the  performance  of 
multiple  individuals”  (Salas,  Cooke,  &  Rosen,  2008,  p.  541).  Van  Berio,  Lowyck,  and 
Schaafstal  (2007)  also  concluded  team  training  must  consider  both  task  work  (primarily  the 
cognitive  and  technical  skills  necessary  to  perform  tasks)  and  teamwork  (primarily  the  social  and 
communicative  skills  required  to  function  within  a  team). 

Cooke,  Salas,  and  Cannon-Bowers  (2000)  summarized  team-based  competencies  into 
three  primary  dimensions:  cognitions,  skills,  and  attitudes.  These  competencies,  while  similar  to 
traditional  task  categorizations  for  individual  performance,  refer  specifically  to  the  team-based 
dimensions  of  cognitions,  skills,  and  attitudes.  For  example,  team  or  collective  cognitions  refer 
to  those  cognitive  tasks  perfonned  as  a  member  of  team,  rather  than  as  an  individual.  The  same 
is  true  of  skills  and  attitudes.  These  competencies  reflect  those  of  the  team  interactions,  rather 
than  focusing  only  on  individual  perfonnance.  Weaver,  Rosen,  Salas,  Baum,  and  King  (2010) 
have  taken  further  steps  to  define  these  competencies  as  reflected  in  Table  2.  Most  competencies 
are  unique  to  team  performance  environments,  and  all  reflect  a  team  orientation.  For  example, 
“accurate/shared  mental  models,”  a  competency  within  cognition,  reflects  the  mental 
organizational  structure  or  model  that  an  individual  has  relative  to  the  role  in  and  interaction  with 
the  team.  Thus,  this  competency  includes  an  individual  level  requirement  (i.e.,  accuracy)  and  a 
collective  requirement  (i.e.,  shared).  Other  competencies,  such  as  mutual  trust,  backup  behavior, 
and  conflict  management,  are  relevant  only  in  a  collective  or  team  environment.  Team-based 
competencies  can  inform  collective  training  requirements  that  may  not  be  identified  from  a 
solely  task-based  analytic  model. 
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Table  2 

Team  Training  Competencies 


Competency 

Category 

Competency 

Descriptor 

Attitude 

Mutual  trust 

Team/collective  efficacy 
Team/collective  orientation 
Psychology  safety 

Trust  across  and  between  team  members 

How  well  the  team  works  together  effectively 

Common  focus  of  the  team 

Feeling  of  security  in  team  and  decision 

Cognition 

Accurate/shared  mental  models 
Cue-strategy  associations 

Common  cognitive  model  for  mission  activities 
“Triggers”  that  provide  cues  to  associate  action 

Behavior 

Closed-loop  communication 

Team  leadership 

Mutual  performance  monitoring 
Backup/supportive  behavior 
Conflict  management 

Mission  analysis 

Team  adaptation 

Communications  within  the  team 

Leadership  roles  for  each  crew  member 

Individual  monitoring  of  team  performance 

Performance  of  the  individual  that  benefits  the  team 
Management  of  disputes  within  team 

Individual,  collective  analysis  of  desired  outcomes 
Ability  of  team  unit  to  adapt  to  any  change 

Adapted  from  Weaver,  Rosen,  Salas,  Baum,  and  King  (2010) 


Conceptualizing  mission  performance  as  composed  of  individual  tasks  and  team 
competencies  leads  also  to  an  alternative  view  of  perfonnance  assessment.  In  this  instance, 
individual  performance  metrics  should  be  supplemented  by  team-based  metrics  in  order  to 
clearly  integrate  team  performance  outcomes  as  a  part  of  mission  perfonnance  assessments.  In 
their  research  measuring  teamwork  within  an  antiaircraft  team,  Smith- Jentsch,  Johnston,  and 
Payne  (1998)  concluded  that  there  are  differences  between  team  dimensions  for  training  and 
team  dimensions  for  assessment.  Based  on  their  research,  assessment  measures  of  team 
competencies  need  to  be  discrete  and  observable,  even  though  that  is  not  necessarily  always  the 
case  for  other  training  tasks,  skills,  or  other  requirements.  Thus,  Smith-Jentsch  et  al.  (1998) 
refined  the  the  team  competencies  they  identified  to  eliminate  redundancies  and  nonobservable 
behavior.  They  also  expanded  the  number  of  training  dimensions  that  their  research  indicated 
were  directly  relevant  to  desired  team  performance  outcomes.  Applying  the  same  approach  to 
our  findings,  we  refined  the  basic  dimensions  summarized  in  Table  2  to  incorporate  assessment 
parameters  supported  by  our  research  review  and  documented  crew/team  training  requirements. 
While  the  “original”  dimensions  may  still  facilitate  the  identification  of  general  team  training 
requirements,  the  refined  dimensions  summarized  in  Table  3  provide  a  more  detailed  foundation 
for  infonning  what  will  be  needed  in  an  FEA  strategy. 
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Table  3 

Teamwork  Dimensions 


Training  Dimension 

Assessment  Dimensions 

Descriptor 

Communication 

Communication 

Ability  of  the  team  to  connect  in  speech  and 
writing 

Feedback 

Information  exchange 

The  way  in  which  communication  occurs 
resulting  in  understanding  by  the  recipient 

Backup  behavior 

Supportive  behavior 

Demonstration  of  the  way  in  which  an 
individual’s  behavior  is  supportive  of  team 
outcomes 

Team  initiative  /  leadership 

Team  initiative  /  leadership 

The  way  in  which  the  individual  crew  members 
and  the  team  take  charge  to  achieve  the  goals  of 
the  team 

Monitoring 

Tracking  of  team  performance  and  outcomes 

Coordination 

Organization  and  synchronization  of  effort 
within  them 

Situational  awareness 

Ability  of  individual  crew  members  to  know 
what  is  going  on  around  them 

Adapted  from  Smith-Jentsch,  Johnston,  and  Payne  (1998) 


Huddlestone  and  Pike  (2008)  recommended  supplementing  task-based  analysis  with 
analyses  that  focus  on  collective  training  requirements.  They  pointed  out  that  collective  tasks 
typically  fall  into  processes  in  the  realms  of  command  and  control,  communication,  and 
teamwork.  Based  on  this  research,  they  developed  a  training  needs  analysis  (TNA)  Triangle 
Model  that  supplements  traditional  task  analyses  with  two  additional  domains:  training 
environment  analysis  and  training  overlay  analysis  (Huddlestone  &  Pike,  2009).  Training 
environment  analysis  examines  real,  synthetic,  and  communications  environments  that  surround 
an  operational  setting.  Training  overlay  analysis  focuses  on  instructor  roles  and  other 
instructional  design  elements.  The  training  environment  analysis  is  particularly  relevant  to  the 
current  research  as  it  emphasizes  the  need  to  identify  the  parties  and  processes  associated  with 
each  training  task.  An  examination  of  the  training  environment  places  the  subject  area  into  a 
mission  context  and  brings  other  individuals  and  organizations,  with  their  associated 
interdependent  procedures  and  processes,  into  the  analytic  framework. 

Collective  training  within  a  team  context  is  not  a  simple  matter  of  joining  individual  tasks 
at  the  proper  time  and  place.  Team  researchers  agree  that  collective  and  team  training  are 
subject  to  different  processes,  attributes,  requirements,  and  goals  than  individual  training.  Lum, 
Fiore,  Rosen,  and  Salas  (2008)  suggested  that  collective  training  for  team  problem  solving  needs 
to  undergo  four  collaborative  stages:  (a)  knowledge  construction,  (b)  collaborative  team  problem 
solving,  (c)  team  consensus,  and  (d)  outcome  evaluation  and  revision.  Knowledge  construction 
is  tied  to  team  perfonnance,  in  general,  with  multiple  researchers  pointing  to  the  importance  of 
shared  cognition  (Salas,  Cooke,  &  Rosen,  2008)  or  shared  mental  models  as  precursors  to 
successful  team  perfonnance.  Shared  cognition  is  the  collective  cognitive  activity  from 
individual  group  members  where  the  collective  activity  impacts  group  goals.  Shared  cognition  is 
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linked  to  knowledge.  Thus,  knowledge  possessed  by  effective  teams  has  been  referred  to  as 
shared  knowledge,  or  shared  mental  models  (Klimoski  &  Mohammed,  1994).  In  their  review  of 
50  years  of  team  performance  research,  Salas  and  his  associates  concluded  that  shared  cognition 
is  a  critical  driver  of  team  performance  (Salas  &  Fiore,  2004,  as  cited  in  Salas,  Cook,  &  Rosen, 
2008).  Shared  cognition  is  particularly  important  in  developing  shared  mental  models,  team 
situation  awareness,  effective  team  communication,  and  team  decision-making. 

Decision-making 

Decision-making  and  problem  solving  are  usually  distinguished  as  separate  processes. 
However,  as  Jonassen  (2012)  pointed  out,  decision-making  is  really  a  core  component  of 
problem  solving.  Problems  can  be  categorized  along  a  continuum  from  well-structured  to  ill- 
structured  (Jonassen,  2000).  As  problems  become  more  ill-defined  or  complex,  the  decision¬ 
making  process,  in  turn,  becomes  more  complex,  encompassing  multiple  types  of  decisions  and 
an  iterative  rather  than  linear  process.  According  to  Yates  and  Tschirhart  (2006),  there  are  many 
different  kinds  of  decisions,  including: 

•  choices,  a  selection  is  made  from  a  subset  from  larger  set  of  alternatives; 

•  acceptances/rejections,  a  binary  choice  (i.e.,  yes  or  no)  in  which  only  one  specific  option 
is  accepted  or  not; 

•  evaluations,  statements  of  worth  backed  up  with  a  commitment  to  act;  and 

•  constructions,  attempts  to  create  ideal  solutions  given  available  circumstances  and 
resources. 

Planning  for  and  executing  military  operations  clearly  require  all  four  types  of  decision¬ 
making,  with  the  decisions  often  made  under  great  stress  and/or  time  pressures.  As  the  decision 
type  becomes  more  complex,  so,  too,  should  its  associated  training.  Standard  FEA  and  task 
analysis  procedures  have  proven  adept  at  determining  training  requirements  for  choices  and 
acceptances/rejections,  as  these  decisions  often  follow  rule-based  procedures  or  steps.  However, 
determining  the  proper  training  requirements  for  evaluation  and  construction  decisions  have 
proven  a  great  deal  more  elusive. 

Decision-making  during  the  planning  and  executing  of  military  operations  can  be  viewed 
as  a  fairly  unique  form  of  decision-making.  While  one  hopes  that  operational  decision-makers 
rationally  consider  an  extensive  array  of  potential  options  and  variables,  it  is  understood  that 
Soldiers  typically  do  not  have  the  time  to  methodically  complete  a  decision-matrix  or  extensive 
risk  assessment  in  the  middle  of  an  operation  or,  as  with  Patriot  fire  control  officers,  to  arrive  at  a 
decision  to  engage  an  indentified  threat.  Thus,  the  process  is  more  along  the  lines  described  by 
Gigerenzer  and  Goldstein  (1996),  who  argued  individuals'  inferences  during  decision-making  are 
frugal  and  do  not  rely  on  much  deliberation.  Mission  planning  intends  to  limit  major  decisions 
during  operations,  yet  there  are  limits  to  what  can  be  reasonably  known,  predicted,  and  planned 
in  any  situation  or  environment.  The  greatest  challenge  for  military  decision-makers  is  not  what 
to  do  when  events  unfold  as  planned  or  anticipated,  but  what  to  do  when  the  unexpected  (and 
unplanned)  occurs  and  more  adaptive  decision-making  that  can  adjust  to  an  evolving,  fluid 
operational  environment  is  needed. 
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Klein,  Calderwood,  and  Clinton-Cirocco  (1986)  examined  decision-making  by  proficient 
personnel  under  conditions  of  time  pressure  where  life  and  property  were  at  stake,  similar  to 
what  is  the  norm  in  military  operations.  The  authors  found  evidence  of  simultaneous 
comparisons  and  relative  evaluation  of  two  or  more  options  in  less  than  12%  of  the  decision 
points.  Using  experience  and  a  pre-established  mental  model  were  found  to  be  the  most  common 
processes  to  assess  the  situation  as  typical  and  to  identify  a  course  of  action.  Based  on  this 
research,  the  authors  proposed  a  Recognition  Primed  Decision  (RPD)  model  that  emphasized  the 
use  of  recognition  rather  than  analysis  for  rapid  decision-making. 

Fadde  (2009)  has  since  applied  the  RPD  model  to  training  environments,  proposing  the 
training  of  recognition  skills  to  hasten  the  development  of  expertise.  RPD  has  been  assumed  to 
require  years  of  applicable  experience.  Fadde,  however,  suggests  expert  recognition 
performance,  which  requires  a  rapid  sequence  of  recognition,  decision,  and  action,  can  be 
accelerated  by  training  recognition  skills.  Fadde  points  out,  “Recognition  training  does  not 
replace  direct  instruction  in  rules,  concepts,  and  procedures  but  rather  enhances  it... at  the 
appropriate  time  in  the  learner’s  development”  (p.  360).  Given  the  demand  for  proficient  Patriot 
crew  members,  it  seems  worth  exploring  the  possiblity  of  integrating  Patriot  specific  “expertise” 
training  into  the  training  continuum. 

Satisficing,  another  decision-making  strategy,  may  also  have  relevence  to  this  discussion. 
Satisficing  is  a  decision-making  strategy  or  cognitive  heuristic  that  entails  searching  through  the 
available  alternatives  until  an  acceptability  threshold  is  met  (Coleman,  2006).  In  a  team 
environment,  the  team  members  search  through  available  choices  to  come  to  consensus  on  the 
best  strategy.  While  the  strategy  itself  may  apply  to  decision-making  in  complex  situations,  it 
also  is  dependent  upon  the  types  of  decisions  to  be  made,  and  the  timing  of  the  decisions. 
Satisficing  is  a  strategy  to  be  considered  when  there  is  an  unlimited  amount  of  infonnation 
available  and  it  is  necessary  to  eliminate  options  (Sternberg,  2009).  In  the  use  cases  to  be 
considered  for  Patriot  training,  the  application  of  this  model  may  be  relevant. 

Expertise 

Training  to  expert  performance  is  not  typically  addressed  in  military  training  FEAs.  This 
may  be  due  to  the  expectation  that  expertise  is  achieved  through  field  experience  and  on-the-job 
training,  two  things  that  do  not  often  fall  within  the  realm  of  an  FEA  training  requirements 
analysis.  It  has  been  pointed  out  that  expertise  can  be  viewed  as  a  learning  process  characterized 
as  an  active  problem  solving  process  in  a  specific  context  (Valkeavaara,  1999).  Furthermore, 
van  Gog,  Erikson,  Rikers,  and  Paas  (2005)  pointed  out  that  expertise  is  not  related  to  the  amount 
of  experience  in  the  domain,  but  rather  the  amount  of  deliberate  effort  and  practice  that  has  been 
applied  to  improve  performance.  According  to  Valkeavaara  (1999),  a  better  understanding  of 
expertise  in  the  field  can  be  achieved  by  taking  a  closer  look  at  problematic  situations 
encountered  by  experts  in  the  field,  yielding  information  about  the  types  of  situations,  the  types 
of  solutions  sought  or  attempted,  and  the  type  of  knowledge  gained  from  experience.  Important 
to  an  FEA  is  the  type  of  knowledge  gained  from  problematic  encounters,  and  what  knowledge 
base  is  needed  to  place  operators  in  a  position  to  resolve  and  leam  from  those  situations.  A 
related  question  is  whether  problematic  situations  (and  accompanying  solutions  or  knowledge 
gained)  can  be  categorized  in  a  way  to  inform  FEA  data  requirements  and  specific  training 
recommendations. 
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Chi  (2006)  identified  a  number  of  expertise-related  concepts  relevant  to  military  settings, 
including  a  proficiency  scale  of  six  expertise  related  categories:  novice,  initiate,  apprentice, 
journeyman,  expert,  and  master.  These  categories  may  be  helpful  in  constructing  a  performance 
continuum  that  aligns  with  training  requirements,  although  the  categories  do  not  align  directly  to 
commonly  used  military  categories.  Of  greater  interest  in  determing  what  defines  an  “expert,” 
Chi  also  described  manifestations  of  experts'  skills.  According  to  Chi,  experts  excel  in  the 
following  ways  (note  that  all  references  are  cited  in  Chi,  2006,  pp.  23-24): 

•  experts  generate  the  best  solutions  in  solving  problems,  typically  faster  and  more 
accurately  than  non-experts  (Klein,  1993); 

•  experts  can  detect  and  recognize  cue  stimuli  faster  than  novices,  as  well  as  underlying 
structures  of  problems  or  situations  (Chi,  Feltovich,  &  Glaser,  1981); 

•  experts  analyze  problems,  particularly  qualitatively,  compared  to  novices  (Simon  & 
Simon,  1978;  Voss,  Greene,  Post,  &  Penner,  1983); 

•  experts  have  more  accurate  self-monitoring  skills  in  tenns  of  their  ability  to  detect  errors 
and  the  status  of  their  own  comprehension  compared  to  non-experts; 

•  experts  are  better  at  choosing  the  most  appropriate  strategies  compared  to  non-experts; 

•  experts  are  more  opportunistic  than  novices,  better  at  utilizing  available  resources 
(Gilhooly  et  ah,  1997);  and 

•  experts  easily  retrieve  relavent  knowledge  and  strategies  (Alexander,  2003,  p.  3)  and 
perfonn  skills  more  automatically  (Schneider,  1985)  than  novices. 

On  the  other  hand,  experts  also  have  shortcomings,  also  identified  by  Chi  (2006): 

•  expertise  is  domain-limited,  and  not  usually  transferable  to  other  domain  areas; 

•  experts  may  be  overconfident  and  overestimate  their  understanding  of  a  situation  (Chi, 
1978,  cited  by  Chi,  2006); 

•  experts  may  gloss  over  a  problem  or  situation  and  fail  to  perceive  or  recognize  details 
attended  to  by  novices; 

•  expertise  is  context  dependent  (i.e.,  domain  knowledge  may  not  be  accessed  by  experts  if 
contextual  cues  of  a  problem  or  situation  are  not  recognized; 

•  experts  are  often  inflexible,  and  not  as  ready  as  novices  to  adapt  to  news  rules  or 
procedures; 

•  experts  are  often  inaccurate  in  predicting  perfonnance  of  others  (particularly  novices)  and 
are  less  likely  than  novices  to  incorporate  peer  feedback  into  their  behavior;  and 

•  experts  are  often  biased  in  their  assessment,  with  the  bias  reflecting  their  knowledge  and 
experience  domain. 

The  above  expert  characteristics  should  be  considered  during  the  course  of  an  FEA, 
particularly  when  collecting  and  analyzing  data  from  subject  matter  experts.  While  experts  can 
provide  the  most  detailed  and  “tested”  infonnation,  they  may  also  misjudge  the  relevance, 
difficulty,  or  training  implications  of  operational  tasks  and  skills.  Furthennore,  while  experts 
may  be  more  proficient  and  capable  of  rapid  decision-making,  other  qualities  (e.g.,  perception 
biases,  overconfidence,  or  inflexibility)  may  negatively  influence  their  demonstrated  expertise 
over  mission  events  in  the  operational  setting. 
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Kozlowski  (1998)  differentiated  routine  expertise  from  adaptive  expertise.  Kozlowski’s 
relationships  between  expertise,  situation  predictability,  and  task  requirements  can  be  depicted 
visually  as  illustrated  in  Figure  5.  According  to  Kozlowski’s  research,  routine  expertise  is 
effective  in  well-defined  predictable  situations,  and  can  be  achieved  through  the  learning  and 
rehearsal  of  routine  tasks.  Adaptive  expertise,  on  the  other  hand,  is  needed  for  success  in  ill- 
defined,  unpredictable  situations.  Adaptive  expertise  requires  problem  solving  and  adaptation  of 
previously  learned  skills,  knowledge,  and  experiences  to  achieve  successful  task  outcomes. 
Routine  and  adaptive  expertise  lies  on  a  continuum  where  the  less  defined  or  predictable  a 
situation  becomes,  the  more  adaptive  expertise  is  needed  to  be  successful.  Conversely,  the  more 
defined  or  predictable  a  situation  is,  the  more  likely  routine  expertise  is  sufficient  for  success. 
Expertise  requirements  (i.e.,  routine  versus  adaptive)  can  be  detennined  by  understanding  the 
situation  and  operating  environment.  If,  for  example,  the  heavy  horizontal  bar  in  Figure  5 
represents  the  predictable  (left  end  of  the  scale)  versus  unpredictable  (right  end  of  the  scale) 
nature  of  the  Patriot  mission  environment,  then  the  white  portion  left  of  the  vertical  bar 
represents  the  realm  of  tasks  and  skills  that  can  be  trained  using  routine  task  rehearsal  while  the 
gray  portion  left  of  the  bar  represents  more  complex  tasks  and  skills  that  require  more  adaptive 
expertise. 


Routine  Expertise 


Adaptive  Expertise 


Rehearsal 


Problem  Solving 
Adaptation 


Figure  5.  Expertise  Continuum. 


Following  our  illustration  in  Figure  5,  it  can  be  theorized  that  some  Patriot  training  and 
operations  concerns  are  associated  with  tasks  and  skills  that  lie  in  the  adaptive  expertise  (gray 
portion  left  of  the  bar).  While  there  is  heavy  reliance  on  checklists  for  task  performance,  the 
areas  in  adaptive  expertise  may  be  required  is  in  those  situations  where  decision-making  in 
uncertain  circumstances  occurs.  If  this  is  confirmed  through  an  adequate  FEA,  then  new  or 
revised  Patriot  training  should  emphasize  the  acceleration  of  adaptive  expertise  required  for 
those  complex  tasks. 


According  to  Kozlowski  (1998): 

Adaptive  expertise  entails  a  deep  comprehension  of  the  conceptual  structure  of  the 
problem  domain.  Knowledge  must  be  organized,  but  the  structure  must  be  flexible.  The 
process  goes  beyond  procedural  knowledge  of  an  automatic  sort.  Adaptive  experts 
understand  when  and  why  particular  procedures  are  appropriate  as  well  as  when  they  are 
not.  A  key  factor  for  the  development  of  adaptive  expertise  is  the  encouragement  of 
active  learning  strategies  during  training,  (p.  119) 
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Hawley  (2009)  recommended  three  strategies  that  apply  Kozlowski's  principles  to  Patriot 
training:  (a)  extensive  deliberate  practice  with  expert  feedback;  (b)  scenarios  characterized  by 
increasing  variability  and  novelty  that  challenge  routine  skills;  and  (c)  a  focus  on  developing 
sense  making  skills  that  facilitate  an  operator’s  ability  to  recognize  when  to  shift  from  automatic 
processing  to  critical  thinking  and  problem  solving  (see  also  Kozlowski,  1998). 

Other  researchers  offer  additional  recommendations  for  training  expertise.  Fadde  (2013) 
proposed  an  instructional  design  theory  of  Expertise-Based  Training  (XBT)  as  a  way  to  train 
expertise  by  decreasing  decision-making  time.  This  approach  is  based  on  the  recognition 
component  of  the  previously  described  RPD  model,  with  a  focus  on  how  to  accelerate  the 
decision-making  training.  Fadde's  strategy  was  designed  to  be  introduced  early  in  the  training 
continuum,  and  can  be  delivered  using  laptops  or  mobile  technologies  to  supplement  and  not 
replace  current  training.  Bills  (2009)  concluded  that  a  “total  training  approach”  to  complex 
training,  including  simulation  and  gaming,  will  advance  metaskill  development  associated  with 
expertise.  Adams  and  Ericsson  (2000)  demonstrated  the  effectiveness  of  problem  focused 
training  for  aviation,  but  point  out  training  effectiveness  varies  between  novices  and  experts. 
Finally,  it  has  been  shown  that  one  effect  of  expertise  is  the  development  of  applicable  schemas, 
which,  in  turn,  reduce  the  working  memory  load  (Van  Gog,  Ericsson,  Rikers,  &  Paas,  2005). 
While  schemas  are  likely  developed  during  Patriot  training,  it  is  unknown  whether  this  schema 
development  is  a  targeted  training  objective  or  an  unintended  consequence  of  incoming  Soldiers’ 
implicit  prior  knowledge  and  experiences. 

Situational  Awareness  and  Automated  Systems 

Situational  awareness  is  a  construct  common  to  military  operation  environments  and  is 
closely  tied  to  Patriot  operational  concerns,  including  decision-making  (Endsley,  1997b; 
Guitouni,  Belanger,  &  Wood,  2011),  supervisory  control  (Hew,  Lewis,  Radunz,  &  Rendell, 
2010),  and  vigilance  (Wann,  Parasuraman,  &  Matthews,  2008).  Situational  awareness  is  a 
critical  input  for,  but  is  separate  from,  mission  and  operational  decision-making.  Situational 
awareness  skills  enable  decision  makers  to  processes  the  context  for  evaluating  events, 
identifying  potential  courses  of  action,  and  determining  appropriate  interventions.  Thus, 
situational  awareness  can  be  conceptualized  from  multiple  perspectives.  Perhaps  the  most 
common  framework  considers  it  in  terms  of  differing  levels  of  awareness.  Endsley  (1997b) 
described  three  levels  of  situational  awareness:  (a)  Level  1,  perception  of  the  elements  in  the 
environment,  (b)  Level  2,  comprehension  of  the  current  situation,  and  (c)  Level  3,  projection  of 
future  status.  Guitouni,  Belander,  and  Wood  (2011)  on  the  other  hand  conceptualized  situational 
awareness  as  one  of  three  types:  (a)  Baseline  or  Routine,  to  provide  monitoring  or  warning 
services;  (b)  Targeted,  focused  on  a  specific  operation  or  event;  and  (c)  Decision  Support,  which 
integrates  infonnation  from  baseline  and  target  situational  awareness  necessary  to  decide  on  an 
intervention. 

Nofi  (2000)  pointed  out  that  situational  awareness  is  not  just  an  individual  construct,  but 
that  it  applies  to  teams  and  collective  environments,  as  well.  Harknett  (1996,  as  cited  in  Nofi, 
2000)  first  advanced  the  idea  of  shared  situational  awareness  as  a  key  manifestation  of 
organizational  change  through  electronic  networking  technologies.  According  to  Nofi,  shared 
situational  awareness  implies  all  team  members  interpret  a  given  situation  in  the  same  way. 
Stanton  et  al.  (2006)  built  upon  this  position  by  insisting  each  team  member's  unique,  but 
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compatible,  situational  awareness  contributes  to  the  shared  situational  awareness  of  the  team. 
Stanton's  model  of  distributed  situational  awareness  seems  more  reflective  of  common  military 
operational  environments  such  as  Patriot,  where  each  crew  member  is  receiving  varying,  distinct 
types  and  amounts  of  information.  Achieving  and  maintaining  effective  collective  or  team 
situational  awareness  is  a  significant  challenge  in  complex  command  and  control  operations. 
Over  time,  operators  and  designers  have  often  turned  to  technology  to  enhance  Soldiers’ 
situational  awareness  or,  at  least,  their  ability  to  quickly  discern  critical  aspects  of  the  situation 
that  require  direct  action  or  interventions.  With  a  greater  capacity  than  their  human  counterparts 
to  near  simultaneously  monitor  and  interpret  an  increasingly  broader  range  of  varied  inputs  from 
multiple  systems  and  to  quickly  act  on  the  calculative  outcomes  of  preset  conditions,  automated 
processes  and  systems  have  grown  in  scope  and  presence  throughout  the  military. 

While  the  proliferation  of  automated  systems  may  be  intended  to  reduce  operator 
workload,  they  also  typically  decrease  operator  situational  awareness  by  lessening  operators’ 
interaction  with  the  system  (Endsley,  1997b).  A  key  aspect  of  situational  awareness,  vigilance, 
is  affected,  perhaps  to  new  levels  of  complacency  due  to  increasing  over-reliance  on  automation. 
Vigilance  failures  significantly  reduce  situational  awareness  as  operators  may  neglect  monitoring 
tasks,  attempt  to  monitor  but  do  so  casually  or  poorly,  or  are  aware  of  indicated  problems  but 
falsely  attribute  any  alarms  to  system  errors  (Endsley,  1997b).  Hew  and  associates  (2010) 
attribute  poor  situational  awareness  in  automated  systems  to  the  fact  that  these  systems  require  a 
different  form  of  situational  awareness  that  may  be  neglected  during  training,  related  to  the 
interaction  of  vigilance  in  situations  with  automated  systems. 

Hawley,  Mares,  and  Giammanco  (2005;  2006),  Hew  (2010),  and  their  respective 
associates  also  agreed  that  automated  systems  present  unique,  unexpected  challenges  to 
situational  awareness.  Based  on  their  research,  these  challenges  are  directly  tied  to  supervisory 
control.  Supervisory  control  is  where  a  machine  closes  a  control-loop,  i.e.  decides  what  action  is 
required  and  executes  it,  and  a  human  supervisor  intennittently  adjusts  the  machine.  The 
differences  between  traditional  human  control  and  supervisory  control  paradigms  are 
summarized  in  the  diagrams  in  Figure  6,  as  adapted  from  Hawley,  Mares,  and  Giammanco 
(2005). 
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Representative  person-machine 
process  control  system 


Basic  supervisory  control  paradigm 


Figure  6.  Comparison  of  supervisory  control  versus  person-machine  control  processes.  Adapted 
from  Hawley,  Mares,  and  Giammanco  (2005) 

As  Figure  6  illustrates,  in  the  supervisory  control  paradigm,  the  operator  is  further 
removed  from  controlling  and  making  decisions  within  the  environment.  Furthennore,  in  the 
supervisory  control  paradigm,  the  system  provides  a  more  advanced  decision  aiding  function 
through  which  the  system  assumes  control  of  the  actions  and  processes  that  results  in  that 
decision  aid.  Hawley  and  his  colleagues  (2005,  pp.  19-21)  raise  a  number  of  concerns  resulting 
from  this  transition  of  controls,  interactions,  and  processes  to  an  automated  system: 

•  Newly  automated  systems  seldom  provide  all  anticipated  benefits.  There  are  often 
discrepancies  between  what  operators  expect,  what  designers  require,  and  actual  system 
perfonnance. 

•  While  monitoring  is  usually  increased,  automation  does  not  decrease  nor  necessarily 
simplify  work  requirements,  but  changes  them. 

•  A  belief  in  a  system’s  infallibility  can  lull  operators  into  a  false  sense  of  security. 
Consequently,  operators  may  not  perform  necessary  checks.  Conversely,  in  systems  that 
frequently  fail,  operators  may  lose  trust  in  the  information  they  receive  from  the  system. 

•  Automation  transfonns  operators  into  system  monitors  rather  than  active  controllers. 

This  is  a  particular  design  problem  in  balancing  control  between  the  operator  and  the 
system. 

•  As  more  operator  control  functions  are  replaced  by  the  system,  the  operator  becomes  less 
familiar  with  how  things  should  function,  potentially  delaying  response  time  to  abnonnal 
situations  or  events. 
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•  Automation  does  not  eliminate  the  possibility  for  human  error,  but  relocates  human  errors 
to  a  different,  often  higher,  level.  These  higher  level  errors  can  be  more  significant  than 
human  errors  in  a  less  automated  system. 

•  Highly  automated  systems  tend  to  reduce  the  frequency  of  crew  member  interactions, 
resulting  in  a  need  to  intentionally  maintain  crew  communication  and  coordination. 

•  The  operation  of  automated  systems  often  requires  more  training,  particularly  at  a  higher 
conceptual  level. 

Hawley,  Mares,  and  Giammanco  (2005)  reported  that  these  issues  can  and  must  be 
addressed  in  the  designs  for  Patriot  training.  Using  a  proven  framework  from  similar 
perfonnance  domains,  Hawley  and  colleagues  cited  a  training  model  for  complex  decision¬ 
making  (see  McPherson  &  Kernodle,  2003)  that  outlines  three  training  areas  necessary  for 
effective  performance:  (a)  declarative  knowledge,  facts  and  rules  governing  the  problem 
situation;  (b)  tactical  knowledge,  analysis,  planning,  self-monitoring,  anticipation,  etc.,  what  to 
do;  and  (c)  procedural  knowledge,  how  to  execute  the  intent  decided  in  step  two. 

Hawley  and  his  colleagues  concluded  that  Patriot  training  emphasizes  areas  (a)  and  (c), 
but  does  not  adequately  address  area  (b).  They  observed  that  the  missing  pieces  involve 
translating  background  knowledge  and  the  current  tactical  situation  into  an  appropriate  course  of 
action.  They  further  argued  that  Patriot  training  must  change  to  include  the  conditions  for 
deliberate  practice,  including  valid  training  scenarios,  effective  after  action  reviews,  and  time  to 
develop  the  necessary  levels  of  expertise  (Hawley,  Mares,  &  Giammanco,  2005,  pp.  25-26). 

As  discussed  earlier  in  this  report,  an  FEA  should  provide  an  analytic  framework  that 
lays  the  foundation  for  training  designs  and  programs.  The  standardized  DOD  task-based  FEA, 
in  general,  and  the  Army’s  corresponding  SAT  process,  specifically,  is  usually  a  sufficient  and 
effective  means  to  determine  most  individual,  procedurally  defined  training  requirements. 
Advantages  of  using  a  task-based  analysis  include  the  fact  that  it: 

•  breaks  down  mission  requirements  into  discrete,  easy-to-manage  parts; 

•  usually  provides  clear  start  and  stop  indications; 

•  provides  a  relatively  easy  translation  of  tasks  into  learning  objectives; 

•  includes  performance  tasks  that  are  relatively  easy  to  measure;  and 

•  relies  upon  standard  and  common  practices. 

Task-based  analysis  alone,  however,  is  less  effective  in  analyzing  complex  and/or 
cognitively  defined  tasks.  It  is  especially  insufficient  for  analyzing  tasks  that  require  adaptation 
or  variation  from  established  standards,  procedures,  or  routine  practices.  Thus,  key  limitations  of 
task-based  training  analysis  include: 

•  an  insufficient  capability  to  address  cognitive  tasks  that  are  difficult  to  deconstruct  and 
analyze; 

•  the  difficulty,  and  often  insufficiency,  of  translating  individual  tasks  into  team  (or 
collective)  tasks;  and 

•  typical  conceptualizations  of  collective  tasks  as  an  aggregation  and/or  alignment  of 
discrete  individual  tasks. 
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Supplementing,  rather  than  replacing,  the  Army’s  SAT  process  with  targeted  analyses  should 
maintain  the  advantages  of  the  current  process,  while  addressing  its  disadvantages  and 
weaknesses  in  regards  to  more  complex  contexts. 

Other  FEA  Approaches 

The  scope  of  training  contexts  and  issues  drive  the  design  of  any  FEA  approach.  Thus, 
the  scope  of  an  FEA  must  be  sufficient  to  ensure  the  questions,  issues,  or  deficiencies  behind  the 
triggering  event(s)  will  be  addressed  sufficiently.  Two  questions  essentially  determine  an  FEA’s 
scope:  (a)  who  to  include  (i.e.,  training  for  what  personnel  category(s));  and  (b)  what  aspects  of 
training  should  be  included  in  the  analysis? 

In  the  case  of  Patriot  training,  the  question,  “Who  should  be  included  in  the  FEA?”  can 
be  derived  directly  from  existing  mission  requirements.  This  is  due  to  the  crew  orientation  of 
mission  performance  as  well  as  the  questions  driving  the  FEA  request.  For  example,  questions 
regarding  the  Patriot’s  mission  are  not  confined  to  individual  crew  positions,  but  include  system 
failures  where  the  system  is  conceptualized  as  both  human  and  technology  systems.  Focusing  on 
one  or  two  individual  crew  positions  would  not  include  the  level  of  system  analysis  required  to 
comprehensively  address  mission  processes  and  outcomes.  Including  all  crew  positions  in  the 
FEA  will  provide  a  system  framework  for  understanding  mission  requirements  that  can  then  be 
translated  into  training  requirements  for  the  team,  as  well  as  for  individual  crew  members. 

Answers  to  the  question,  “What  aspects  of  training  should  be  included?”  can  be  derived 
directly  from  the  issues  driving  the  FEA  requirement.  Based  on  published  reports  of  Patriot 
mission  performance  and  training,  or  apparent  shortfalls  in  these  areas  (see  Hawley,  2007  & 
2011),  an  FEA  should  include  a  careful  consideration  of  the  various  levels  of  expertise  required 
during  operational  missions.  Furthermore,  concerns  expressed  in  these  reports  about  individual 
and  team  adaptability,  interpretation  of  information,  and  decision-making  effectiveness  during 
mission  events  suggest  that  team  training  requirements  need  to  be  included  in  the  FEA  process. 
Thus,  the  entire  training  continuum  needs  to  be  considered,  from  initial  Patriot  training  through 
expert  performance  (see  Figure  7). 


Untrained 


Expert 


▲ 


Milestones 


What  needs  to  occur  between  milestones? 


Training 


Experience 


Figure  7.  Training  continuum. 


25 


Critically  applying  this  training  continuum  in  the  FEA  will  emphasize  the  need  to 
identify  a  wide  range  of  training  tasks  and  requirements  from  untrained  to  expert  operators.  This 
information  can  then  be  used  to  identify  appropriate  training  milestones  and  the  type  of  training 
required  to  achieve  these  milestones.  Training  recommendations,  such  as  how  to  train,  where  to 
train,  when  to  train,  and  who  to  provide  the  training  to,  can  then  be  overlaid  onto  the  training 
continuum.  This  comprehensive  approach  will  minimize  assumptions  about  when,  how,  and 
who  is  responsible  for  training  across  the  continuum.  It  should  also  provide  a  more  holistic 
approach  to  understanding  individual  and  collective  team  training  progression. 

Collective/Team  FEA 

The  team-based  nature  of  Patriot  batteries  and  crews  dictates  that  training  should  include 
a  strong  collective  emphasis.  This  training  must  focus  on  team  competencies,  collective  tasks 
and  team  processes  to  enhance  team  perfonnance  and  develop  needed  expertise  throughout  the 
team  (see  Salas,  Cooke,  &  Rosen,  2008).  Training  strategies  to  improve  team  members’ 
understanding  of  each  other's  roles,  CRM  and  team  processes,  adaptation,  and  decision-making 
should  be  incorporated  into  crew  members’  training  cycles.  Thus,  the  how,  when,  and  by  whom 
questions  that  drive  training  designs  must  be  addressed  through  an  FEA  process  that 
intentionally  accounts  for  and  analyzes  team  training  requirements.  This  is  important  for  several 
reasons: 


•  “team”  training  requirements  may  be  different  than  individual  or  collective  task  training 
requirements; 

•  team-based  competencies  need  to  be  integrated  into  mission  and  context  specific  training; 
and 

•  perfonnance  should  be  assessed  at  the  team  level,  as  well  as  individual  levels,  as  part  of 
any  evaluation  process. 

Anticipated  benefits  and  risks  of  supplementing  the  standard  FEA  process  with  a  focused 
collective/team  analysis  are  summarized  in  Table  4. 


Table  4 

Collective/T earn  Analysis  Benefits  and  Risks _ 

Benefits  Risks 


Team  training  and  performance  requirements  will  be 
identified 


Individual  and  collective  tasks  will  be  analyzed  within  a 
mission  context 

Integrity  of  the  Army  task-based  FEA  strategy  (i.e., 
SAT)  is  maintained 

Analysis  will  complement  current  individually  focused 
analyses 


Could  de-emphasize  consistent  focus  on  tracking 
individual  progress  within  a  collective  setting;  must 
ensure  that  individual  requirements  and  metrics  are 
maintained  and  augmented  by  collective/team  based 
metrics 

Appropriate  collective/team  metrics  may  not  exist  and 

may  be  difficult  to  develop  within  existing  weapon 

systems  and  doctrines 

Results  may  require  adjustment  of  training 

responsibilities 

Results  may  require  additional  training  resources 
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Team  analyses  should  be  integrated  into  the  Army’s  established  SAT  analysis  process  as 
illustrated  in  Figure  8.  These  analyses  should  be  conducted  immediately  following  the  mission 
analysis,  feeding  directly  into  collective  tasks  analyses  (to  determine  team  task  and  performance 
requirements)  and  individual  job  analyses  (to  determine  individual  training  requirements). 
Individual  and  collective  task  analyses,  in  turn,  highlight  aggregated  training  requirements  for 
the  team,  as  well  as  those  for  individual  crew  members.  The  difference  between  this  approach 
and  the  standard  Anny  SAT  approach  is  the  intentional  focus  and  inclusion  of  collective  and 
team  training  requirements  as  the  main  analytic  framework,  rather  than  identifying  team 
requirements  by  simply  combining  individual  tasks  and  requirements  into  collective 
requirements. 


From  TRADOC  Pam  350-70-6  |  Alt  FEA  Process  Revisions  | 

Figure  8.  Integrating  team  analysis  into  the  U.S.  Army’s  SAT  analysis  process. 

Expert-based  FEA 

Determining  training  requirements  beyond  rote  performance  of  routine  or  clearly  defined 
procedures  requires  an  analysis  of  expert  perfonnance,  or  expertise.  Expertise  analysis  will 
reorient  the  scope  of  training  analysis  by  identifying  the  tasks,  processes,  and  responsibilities  that 
are  necessary  to  respond  effectively  to  unpredicted  or  novel  situations  and  events  characteristic 
of  complex  systems  and  ambiguous  operating  environments.  Expertise  analysis,  by  definition, 
provides  insight  to  advanced  mission  performance  skills,  such  as  situational  awareness  and 
adaptive  decision-making,  that  could  be  de-emphasized  in  a  traditional  task-based  FEA.  An 
expertise  analysis  must: 

•  focus  on  tasks  necessary  to  address  novel  situations  and  events; 

•  extend  beyond  predicable  performance  tasks  to  identify  what  happens  when  the  unknown 
is  encountered;  and 

•  consider  the  training  continuum  of  all  crew  positions. 
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Anticipated  benefits  and  risks  of  supplementing  the  standard,  task  based  FEA  process 
with  a  focused  expertise-based  analysis  are  presented  in  Table  5. 


Table  5 

Expertise  Analysis  Benefits  and  Risks 


Benefits 

Risks 

Increased  likelihood  that  complex  tasks  can  be  deconstructed 
to  inform  training  recommendations 

In-depth  examination  of  distinguishing  characteristics  of  and 
progression  from  novice  to  expert  operator 

Integrity  of  the  Army  task-based  FEA  is  maintained 

Analysis  will  complement  standard  task-based  analyses 
Mission  environment  and  performance  is  primary  analytic 
context 

Recommendations  may  require  a  redistribution  of 
training  responsibilities  across  the  training 
continuum  of  all  crew  members 

Results  may  require  additional  training  resources 

Generally,  the  expertise  analysis  process  would  be  the  same  as  the  standard  SAT  analysis 
process  (see  Figure  3).  The  difference  between  the  standard  SAT  process  and  this  analysis  is  in 
the  scope  of  tasks  and  skill  levels  identified  and  analyzed. 

Event-based  FEA:  Mission  Deconstruction 

Task-based  analysis  following  the  Anny’s  SAT  analysis  process  may  be  insufficient  to 
analyze  all  of  the  collective  and  individual  tasks  associated  with  increasingly  complex,  highly 
automated  weapon  systems  (e.g.,  Patriot  and  AIAMD  mission  performance).  Task-based 
analysis  deconstructs  mission  requirements  into  associated  duties  and  tasks.  Thus,  specific  tasks 
are  the  primary  unit  of  analysis  following  this  process.  An  alternative  approach  deconstructs 
these  mission  requirements  into  other  units  of  analysis,  such  as  the  sequence  of  events  that  are 
required  to  fulfill  mission  requirements  (see  Figure  9).  Event-based  deconstruction  would  drive 
the  identification  of  mission  events,  phases,  critical  milestones,  and  critical  actions. 
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Figure  9.  Event-based  mission  deconstruction. 

Event-based  deconstruction  would  still  follow  mission  requirements;  the  difference  is  in 
the  type  of  analytic  framework  underlying  the  analyses.  Event  analysis  is  based  on  a  systems 
analytic  approach  that  would  consider  the  interdependencies  among  crew  responsibilities  within 
a  required  sequence  of  tasks,  as  well  as  discrete  individual  requirements  during  mission 
perfonnance.  As  illustrated  in  Figure  9,  a  mission-event  deconstruction  would  capture  collective 
task  requirements  within  the  primary  analytic  framework  at  the  beginning  of  the  analysis,  rather 
than  toward  its  end,  which  is  a  common  consequence  of  deriving  collective  requirements  by 
combining  individually  defined  requirements.  This  approach  could  also  potentially  capture 
complex  perfonnance  requirements  and  sequences  that  are  difficult  to  capture  when  relying  on 
individual  task  analysis  results  alone.  Event  analysis  potentially  addresses  several  task-based 
analysis  shortcomings: 

•  task  and  skill  performance  would  be  analyzed  within  the  context  of  mission  perfonnance 
and  requirements; 

•  crew  member  participation  would  be  analyzed  within  the  mission’s  context;  and 
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•  task  complexity  would  not  necessarily  be  over  simplified  during  the  deconstruction 
process. 

Table  6  summarizes  the  anticipated  benefits  and  risks  of  employing  an  event-based 
approach  within  an  FEA. 


Table  6 

Event  Analysis  Benefits  and  Risks 


Benefits 

Risks 

Event-based  team  performance  measurement  would 
minimize  complex  task  analysis  weaknesses 

Incorporates  team/collective  requirements  identification  as 
an  integral  part  of  the  process 

Team  performance  standards  more  readily  identified 

Mission  environment  and  performance  is  primary  analytic 
context 

Task  delineation  may  be  lost 

Data  and  analysis  complexity  may  not  be  repeatable 

Event  analysis  would  not  necessarily  complement  the  Anny’s  standard  SAT  approach. 
Mission  deconstruction  using  events  rather  than  jobs  and  duties  would,  in  fact,  replace  a 
foundational  element  of  the  SAT  process  -  task  analysis.  As  Figure  10  illustrates,  training 
requirements  are  identified  directly  from  event  analysis  findings.  This  approach,  then,  would 
lose  the  task  delineation  that  is  the  basis  of  current  Anny  training  designs  such  as  Patriot. 


Figure  10.  Event  analysis  process. 


Conclusions 

Each  of  the  proposed  FEA  strategies  reviewed  in  this  report  have  distinct  attributes  that 
would  frame  how  they  might  be  adapted  to  augment  the  Army’s  current  SAT  FEA  process  to 
address  the  demands  and  challenges  found  in  complex,  highly  automated  systems  and 
architectures,  such  as  Patriot  and  AIAMD.  Each  strategy  promises  benefits  as  well  as  risks 
compared  to  the  current  SAT  analysis  process.  Table  7  compares  the  benefits  and  risks  of  each 
alternative  with  those  associated  with  the  Anny’s  established  SAT  process. 
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Table  7 

Alternative  Comparison:  Benefits  and  Risks 


Army 

Event 

Team 

Expertise 

Benefits 

SAT 

Analysis 

Analysis 

Analysis 

Compatible  with  the  Army’s  SAT  analysis  process 

X 

X 

X 

Utilizes  Army’s  SAT  task-based  approach 

X 

X 

Team  training  requirements  are  a  unit  of  analysis,  emphasizing 

team  performance  requirements  as  complement  to  individual 

X 

X 

performance  requirements 

Increases  the  likelihood  that  complex  tasks  will  be  deconstructed 

Y 

Y 

appropriately 

A 

A 

Identifies  distinguishing  characteristics  of  novice  to  expert 

x 

progression 

Team  performance  standards  more  readily  identified 

X 

Mission  environment  and  performance  is  primary  analytic  context 

X 

X 

X 

Risks 

Results  may  require  adjustment  of  training  responsibilities 

X 

X 

Results  may  require  additional  training  resources 

X 

X 

X 

X 

May  de-emphasize  individual  performance  and  progress 

X 

X 

Appropriate  team  metrics  may  not  exist  or  be  readily  defined 

X 

X 

Process  complexity  may  not  be  repeatable  with  available  FEA 

x 

resources 

The  overriding  conclusion  that  emerges  from  these  comparisons  and  our  review  of 
relevant  literature  and  training  reports  is  that  existing  FEA  approaches  have  some  fairly  well 
recognized  limitations.  It  is  apparent  that  new  FEA  strategies  and  approaches  that  build  upon  the 
strengths  of  multiple  approaches  are  needed  to  address  the  emerging  issues,  requirements,  and 
roles  demanded  by  new  and  evolving  increasingly  complex,  highly  automated  systems  and 
architectures.  Thus,  FEA  strategies  and  designs  that  are  hybrids  of  multiple  approaches, 
integrating  FEA-related  processes,  concepts,  and  analyses,  appear  to  be  the  most  effective  way  to 
address  command  dependencies  within  and  between  echelons,  critical  decision  points,  and 
training  requirements  for  complex,  highly  automated  multisystem  architectures  (e.g.,  the 
emerging  AIAMD  environment),  yet  remain  flexible  enough  to  be  applied  to  current  systems 
(e.g.,  Patriot). 

Two  Alternative  FEA  Approaches 

Based  upon  our  theoretical  review  and  consideration  of  the  Patriot  weapon  system  and  its 
pending  transition  to  the  AIAMD/IBCS  operational  architecture  as  an  exemplar  of  evolving 
complex,  highly  automated  systems,  we  offer  two  alternative  FEA  approaches  that  incorporate 
previously  discussed  aspects  of  event,  expertise,  and  collective/team  analyses.  Both  alternatives 
include  team  analyses  due  to:  (a)  identified  concerns  about  increased  demands  on  crew  and  team 
integration,  perfonnance,  adaptability,  and  decision-making  during  mission  performance;  and  (b) 
the  fact  that  complex  weapons  systems  and  multisystem  architectures  require  collective,  as  well 
as  individual,  participation  from  multiple  team  members  and  crews/teams  to  accomplish  the 
mission.  The  two  recommended  alternatives  are: 
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1 .  Alternative  1 :  Replace  the  standard  SAT  analysis  with  a  combination  of  team  and  event 

analyses. 

2.  Alternative  2:  Supplement  the  standard  SAT  analysis  with  a  combination  of  team  and 

expertise  analyses. 

Alternative  1  reflects  a  departure  from  the  established  task-based  analyses  underlying  the 
Army’s  SAT  FEA  approach.  Rather  than  deconstructing  mission  requirements  into  discrete 
duties  and  tasks,  this  approach  focuses  on  the  sequence  of  events  required  to  fulfill  mission 
requirements.  A  key  aspect  of  this  focus  is  that  it  considers  not  only  broad  mission  requirements, 
but  also  the  sequences  of  individual  and  collective  actions  and  events  that  must  be  accomplished 
to  achieve  defined  mission  milestones  and  critical  actions. 

One  advantage  of  this  alternative  is  that  event  and  team  analyses  complement  each  other, 
especially  in  more  complex  and/or  multisystem  architectures  that  often  demand  an  in-depth 
understanding  of  the  sequence  of  events  and  combination  of  individual  and  collective  actions 
that  frame  critical  mission  requirements,  actions,  and  milestones.  Another  advantage  of  this 
approach  is  that,  being  significantly  different  from  traditional  task-based  analyses,  it  should  lead 
to  a  complete  reconsideration  of  mission  requirements  and  performance  standards  in  tenns  of 
supporting  events,  sequences,  and  roles  rather  than  relying  on  a  simple  or  subjective 
amalgamation  of  related  tasks  to  define  mission  requirements  and  critical  training  areas. 

However,  the  uniqueness  of  an  event-based  approach  may  result  in  data  and  findings  that 
are  unfamiliar  to  personnel  used  to  interpreting  the  results  from  traditional  task-based  FEAs. 
Another  disadvantage  is  that  the  similarity  of  event  and  team  data  and  analyses  may  result  in 
redundant  considerations  and  an  inefficient  process.  There  is  also  the  disadvantage  that 
incompatibility  of  event-analysis  with  task-based  analysis  requires  this  approach  to  replace, 
rather  than  supplement  the  standard  SAT  analysis.  This  effect  may  yield  results  that  are  difficult 
to  translate  into  familiar  training  design  language  and  recommendations. 

Unlike  the  first  alternative,  Alternative  2  centers  on  an  expertise  analysis  that  aligns 
directly  with  the  standard  task-based  analysis  approach.  Alternative  2  combines  the  outcomes 
from  the  traditional  task-based  analyses  with  a  critical  examination  of  corresponding  expertise 
progression  and  requirements  and  collective/team  responsibilities.  Thus,  this  alternative 
expands,  rather  than  replaces,  the  Anny’s  established  SAT  approach. 

Since  Alternative  2  represents  an  expansion  of  current  practices,  it  is  expected  that  the 
associated  FEA  procedures,  data,  and  findings  will  be  more  familiar  to  and  easily  interpreted  by 
potential  users.  Much  like  combining  events  and  team  analysis  strategies  in  the  first  alternative, 
this  alternative  also  has  the  advantage  that  collective/team  and  expertise  analyses  data  and 
findings  will  complement  each  other.  Expertise  analyses  will  extend  what  is  known  or 
detennined  about  individual  and  collective/team  training  tasks,  while  collective/team  analyses 
will  highlight  broader  system  and  crew  requirements.  Of  course,  this  approach  assumes  that 
team  analyses  are  conducted  in  terms  of  defined  mission  perfonnance  requirements. 
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Recommended  FEA  Approach 

Given  the  challenges  of  coordinating  and  integrating  inputs  and  actions  from  multiple 
systems  and  increasing  levels  of  automated  controls  and  decision  processes  associated  with 
evolving  systems,  as  well  as  recent  related  concerns  and  issues  identified  within  Patriot  testing, 
training,  and  operations,  Alternative  2  seems  to  offer  greater  potential  at  this  time  by  integrating 
collective/team  and  expertise  analyses  into  the  standard  SAT  analysis  process.  This  alternative  is 
recommended  for  additional  testing  and  refinement  due  to  (a)  its  compatibility  with  the  standard 
SAT  analysis  process,  and  (b)  the  complementary  nature  of  the  two  analyses.  Additionally, 
including  analyses  targeting  collective  communication  and  coordination  requirements,  as  well  as 
individual  and  collective  task  perfonnance  with  an  examination  of  related  expertise  levels  and 
progression,  should  provide  a  sound  foundation  for  identifying  potential  options  that  address  the 
emerging  issues  within  Patriot  training  and  accelerate  the  development  of  expertise  needed  to 
meet  the  challenges  of  more  complex,  highly  automated  systems  operating  in  frequently 
ambiguous  situations.  Given  the  limitation  of  time  and  resources  for  testing  and  validating  this 
recommended  FEA  strategy,  it  is  also  recommended  that  air  battle  management  tasks  and 
processes  be  the  focus  of  the  next  phase  of  this  research  and  include  an  examination  of: 

•  collective  training  requirements  determined  through  the  team  analysis  portion,  focusing 
on  team  processes,  interdependencies,  management,  and  coordination; 

•  complex  individual  tasks,  particularly  tasks  related  to  adaptability,  decision-making,  and 
problem  solving  determined  through  an  expertise  analysis;  and 

•  the  entire  training  continuum,  by  identifying  training  and  assessment  milestones, 
requirements,  and  crew/system  responsibilities,  including  determining  collective  and 
expertise  task  requirements  from  basic  to  expert. 

The  recommended  approach  will  be  designed  to  align  and  integrate  with  the  U.S.  Anny’s 
standard  SAT  analysis  process  as  shown  in  Figure  11.  Note  that  the  team  analysis  flows  directly 
from  the  mission  analysis,  and  preliminary  team  analysis  findings  are  then  used  to  examine  both 
collective  and  individual  tasks.  It  is  anticipated  that  the  expertise  and  team  analyses  will  inform 
each  other,  and  that  data  collection  procedures  will  leverage  SME  participation  by  integrating 
interview  and  discussion  topics.  Consequently,  while  the  process  illustrated  in  Figure  1 1  is 
shown  to  be  linear  in  nature,  in  practice  it  is  likely  to  be  reiterative  wherein  each  analysis  informs 
and  validates  the  other.  The  process  flow  highlights  the  analytic  priorities  within  the  strategy,  in 
that  collective/team  processes  and  requirements  will  be  used  to  frame  the  context  of  subsequent 
analyses. 
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Figure  11.  Recommended  FEA  approach  process. 

The  team  analysis  is  process  and  function  oriented.  This  analysis  examines  the  functions, 
inputs,  and  outputs  of  each  crew  position  to  identify  roles  and  interdependencies  within  the  crew- 
based  system.  Interdependencies  are  examined  to  identify  the  communication  and  coordination 
processes  and  requirements  within  the  crew  as  well  as  between  the  crew  and  higher  command 
echelons.  Data  to  support  these  analyses  is  collected  through  both  crew  and  individual 
interviews  from  experienced  operators,  team  leaders,  and  unit  leaders.  Interviews  should  begin 
by  framing  the  mission  parameters  then  spiraling  down  into  specific  actions  and  processes. 

Questions  guiding  the  team  analysis  are  oriented  around  teamwork  and  team  perfonnance 
during  mission  performance,  including  the  context,  environment,  team  composition  and 
processes,  and  individual  and  collective  task  requirements,  management,  and  coordination.  The 
following  sample  questions  and  topics  form  the  basis  of  individual  and  crew-based  interviews: 

1 .  Identify  and  describe  the  crew  composition. 

2.  Describe  the  mission  requirements  and  processes. 

3.  Identify  and  describe  key  events/moments  (e.g.,  milestones)  during  each  phase. 

4.  Using  the  defined  mission  requirements,  identify  crew  member  tasks  and  responsibilities. 

5.  Describe  each  crew  member’s  responsibilities  during  each  mission  phase. 

6.  Describe  what  each  crew  member  does  to  accomplish/reach  each  milestone. 

7.  What  are  the  knowledge,  skills,  and  abilities  each  crew  member  needs  to  accomplish  their 
job? 

8.  What  are  the  crew  leaders’  responsibilities  during  each  mission  phase? 

9.  Describe  what  the  crew  does  to  reach/accomplish  each  milestone. 

10.  Describe  how  the  crew  interacts  during  each  phase. 
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1 1 .  Who  coordinates  crew  interactions?  Always?  Does  anyone  else  ever  step  in  to  assist  or 
correct  what  is  happening? 

12.  What  are  the  risks  during  each  mission  phase? 

13.  How  are  risks  mitigated? 

14.  Describe  the  mission  planning  /  preparation  process. 

15.  How  are  errors  detected? 

16.  How  are  errors  corrected? 

The  expertise  portion  of  our  proposed  FEA  begins  with  the  identification  of  the  complex 
tasks  that  will  drive  the  analysis.  Complex  task  identification  is  accomplished  using  two 
methods.  One  method  analyzes  existing  task  lists  and  archival  perfonnance  data.  An 
examination  of  past  performance  data  will  identify  difficult  tasks  as  evidenced  by  lower  than 
average  scores  during  trainee  and  operator  evaluations.  Existing  tasks  are  assessed  for  task 
complexity  as  they  emerge  through  individual  and  crew-based  interviews.  Interviews  and 
examinations  of  standard  operating  procedures  provide  an  opportunity  to  identify  critical 
decision  and  coordination  points,  and  inform  predictable  and  unpredictable  situations.  The 
second  method  seeks  to  identify  “new”  tasks  and  situations  through  SME  interviews.  These 
interviews  should  explore  past  situations  encountered  by  SMEs  that  were  novel  or  unexpected 
based  on  their  training  experience.  These  situations,  actions,  processes,  and  outcomes  are 
documented,  categorized,  and  analyzed  to  identify  common  and  uncommon  situations  and  tasks 
encountered  by  expert  operators.  The  following  sample  questions  and  topics  should  guide  the 
identification  of  “new”  tasks  not  accounted  for  in  existing  task  lists: 

1 .  Identify  and  describe  unpredictable  events  you  have  encountered  during  operations. 

2.  How  were  those  events  initially  identified? 

3.  How  were  those  events  handled/resolved? 

4.  What  critical  knowledge  and/or  skill  did  you  require  to  handle/resolve  the  event? 

5.  What  tasks  and  perfonnance  qualities/characteristics  distinguish  expert  perfonnance  from 
non-expert  performance? 

Next,  the  infonnation  gained  from  SME  interviews  will  be  analyzed  to  translate  broad 
mission  events  into  specific  cognitive  and  perfonnance  tasks.  This  requires  that: 

1 .  mission  events  be  categorized  and  analyzed; 

2.  responsive  behaviors  be  identified,  analyzed,  and  categorized;  and 

3.  training  tasks  be  identified  or  developed  from  these  findings,  then  validated  by  SMEs. 

4.  An  appropriate  sequence  for  individual  and  collective  training  events  will  be  detennined 
with  the  assistance  of  system  experts  and  validated  by  SMEs. 

Expert  operators  and  experienced  team  leaders  will  be  critical  data  sources  for  this  research  since 
the  experience  of  resolving  unpredictable  situations  is  key  to  identifying  the  successful 
application  and  resolution  of  complex  tasks,  such  as  problem  solving  and  decision-making,  in 
operational  settings. 

Our  proposed  FEA  is  designed  to  also  address  the  current  and  proposed  training 
continuum  across  crew  positions,  including  the  timeline  of  current  training  progression  and 
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milestones,  training  requirements  and  recommendations  between  those  milestones,  anticipated 
resource  requirements,  and  recommendations  of  where  training  should  occur  and  who  (i.e.,  the 
institution  or  unit)  should  provide  the  training.  Additional  data  will  focus  on  defining  the  current 
training  situation,  identifying  training  gaps,  and  providing  a  framework  for  later  FEA 
recommendations.  Guiding  questions  and  topics  for  the  training  situation  should  be  oriented  on 
what  is  trained  and  how  it  is  trained,  as  well  as  gaining  organizational  input  about  known 
training  issues.  Preferred  data  sources  for  these  topics  include  command-level  SMEs,  training 
directors,  and  training  documentation  such  as  course  materials,  training  guidance,  and  incident 
reports. 


The  following  sample  questions  are  typical  of  those  used  to  define  the  FEA’s  framework 
and  infonn  the  currrect  training  situation: 

1 .  What  are  known  gaps  between  the  current  training  system  and  mission  requirements? 

2.  What  are  known  problems  with  current  training  system? 

3.  What  are  the  causes  of  those  problems? 

4.  What  are  the  impacts  of  those  problems? 

5.  What  are  the  organizational  objectives  of  the  FEA? 

6.  What  constraints  should  be  accounted  for  during  the  FEA? 

7.  What  individual  KSAs  (Knowledge,  Skills,  and  Abilities)  are  currently  trained? 

8.  How  are  individual  KSAs  currently  trained? 

9.  What  crew  competencies  are  currently  trained? 

10.  How  are  crew  competencies  currently  trained? 

1 1 .  What  crew-specific  leader  KSAs  are  currently  trained? 

12.  How  is  crew-specific  leadership  currently  trained? 

13.  What  is  the  training  progression? 

14.  What  are  the  training  resources? 

The  recommended  FEA  approach  includes  analyses  and  data  to  examine  and  define 
complex  tasks  and  collective  communication  and  coordination  requirements.  It  integrates 
collective/team  analysis  and  expertise  analyses  to  supplement  current  task-based,  individually 
focused  training  requirements.  The  proposed  approach  complements  the  Army’s  established 
SAT  analysis  process,  which  should  enhance  the  ability  of  findings  and  recommendations  to  be 
successfully  integrated  within  future  training  designs,  development  approaches,  and 
implementation  strategies. 

During  the  next  phase  of  this  research,  the  proposed  FEA  strategy  will  be  applied  to  the 
use-case  of  the  Patriot  missile  system.  These  results  will  infonn  any  revisions  necessary  to 
ensure  that  the  selected  FEA  approach  addresses  the  key  complexities  of  the  mission  operations 
in  this  circumstance.  In  the  final  phase  of  this  research,  the  subsequently  revised  approach  will 
then  be  applied  to  a  second  use  case,  i.e.  the  transition  of  Patriot  operations  into  the  AIAMD 
architecture.  The  ultimate  objective  of  this  research  is  to  provide  an  FEA  strategy  uniquely 
attuned  for  defining  the  training  requirements  for  complex,  highly  automated  systems. 
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Acronyms 


4C/ID 

Four-Component  Instructional  Design 

ABM 

Antiballistic  Missile 

ADA 

Air  Defense  Artillery 

ADDIE 

Analyze,  Design,  Develop,  Implement,  and  Evaluate 

AIAMD 

Army  Integrated  Air  and  Missile  Defense 

AMD 

Air  and  Missile  Defense 

AMG 

Antenna  Mast  Group 

ARI 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

BCP 

Battery  Command  Post 

CRM 

Crew  Resource  Management 

CTA 

Cognitive  Task  Analysis 

DOD 

Department  of  Defense 

DSB 

Defense  Science  Board 

ECS 

Engagement  Control  Station 

FCOE 

Fires  Center  of  Excellence 

FEA 

Front  End  Analysis 

HQDA 

Headquarters,  Department  of  the  Anny 

IBCS 

Integrated  Battle  Command  System 

ICC 

Information  and  Coordination  Center 

IDA 

Institute  of  Defense  Analyses 

IFCN 

Integrated  Fire  Control  Network 

ISD 

Instructional  Systems  Design 

KSAs 

Knowledge,  Skills,  and  Abilities 

MANPRINT 

Manpower  &  Personnel  Integration 

MIL-HDBK 

Military  Handbook 

PADDIE 

Planning,  Analysis,  Design,  Development,  Implementation,  and 
Evaluation 

PATRIOT 

Phased  Array  Tracking  Radar  to  Intercept  on  Target 

RPD 

Recognition  Primed  Decision 

SAT 

Systems  Approach  to  Training 
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SME 


Subject  Matter  Expert 


TNA 

TRADOC 

USAADASCH 

XBT 


Training  Needs  Analysis 

U.S.  Army  Training  and  Doctrine  Command 

U.S.  Army  Air  Defense  Artillery  School 

Expertise-Based  Training 
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